BERT: Pre-training of Deep Bidirectional Transformers

Devlin et al., 2018

Fill-in-the-blank on the whole internet. The attention page ends with "remove the mask, that's BERT". This is the rest of that sentence.

Download bert_from_scratch.py — everything below runs from it. NumPy, plus a tiny CPU PyTorch model for the training step.

I'd fine-tuned BERT for a classifier before I understood what pre-training had done to it. Load bert-base-uncased, add a head, train three epochs, ship. What nobody told me is that the paper is one exam question I sat at age nine, plus the encoder you already have with one argument deleted.

Pre-training is a cloze test

You did this at school:

sentence = "the cat sat on the mat"
question = "the cat ___ on the mat"     # the blank
answer   = "sat"                        # the teacher has the original

BERT is that, over 3.3 billion words, with a transformer encoder as the pupil and the loss graded only on the blanks: 

for sentence in the_internet:
    hidden, answers = hide_15_percent(sentence)   # blank some tokens
    guess = encoder(hidden)                       # every token sees BOTH sides
    loss  = cross_entropy(guess, answers)         # marked on the blanks only

That is pre-training: no task in mind, just what tends to sit between "the cat" and "on the mat". Then, per task, bolt a small head on and train the whole thing a little more. That is fine-tuning. Same library, a different main() per task.

TermMeansProgrammer's analogy
MLMMasked language model: hide tokens, predict themA cloze test
BidirectionalEach token attends to its left and its rightNo causal mask. Just mask=None
Pre-train / fine-tuneLearn the language once; then add a head and train every parameter per taskOne shared library, a different main() per task
[CLS]An extra first token whose output stands for the whole inputA reduce() over the sequence into one struct

Why it beat GPT-1 and ELMo

A normal language model predicts token 5 from tokens 1–4, then 6 from 1–5. Every token sees only its left. GPT-1 was exactly that. Fine for generating text, but "bank" in "the bank of the river" has its best clue on the right. So why not let every token see both sides? Because it cheats: in a stacked encoder, token 5 in layer 2 reads token 6 from layer 1, which has already read token 5. The paper says each word could "indirectly see itself". ELMo dodged this with one left-to-right LSTM and one right-to-left, glued at the end: a "shallow concatenation". BERT's one trick: if the token is hidden, there's nothing to leak. Replace it with [MASK], let everyone look everywhere, and the only way to score is to infer it from both sides. The cloze test is not a cute framing. It is what makes deep bidirectionality legal.

The one formula

LMLM = − Σi ∈ M log p(xi | x̃)

x is the original tokens, x̃ the same sequence with 15% of it tampered with, M the set of tampered positions. Read it as stages:

StepShapeWhat it does
x(L,)Token ids of one sequence. L is 512 in the paper, 17 in my script
x̃ = corrupt(x)(L,)Pick 15% of positions. Of those, 80% become [MASK], 10% a random token, 10% untouched
encoder(x̃)(L, d)The attention stack from the attention page, with mask=None. Every row sees every column
head(L, V)One linear layer to vocabulary logits: "which token was here?"
−log p at i ∈ MscalarCross-entropy, but only at the tampered positions. The other 85% are context, not questions

The masking recipe: 15%, then 80/10/10

The odd part is the 80/10/10. Why not replace every picked token with [MASK]? Because [MASK] never appears when you fine-tune. The 10% random and 10% kept-as-is mean the model is sometimes marked on a visible token, so it can never fully trust its input. Train without them and accuracy on a visible token falls from 99.3% to 53.0%: a row in the break-it table, and an ablation in the paper's appendix.

The recipe in NumPy, one sentence and then ten thousand:

original : [CLS] d D e E g G h H a A b B g G h H
corrupted: [CLS] d D e _ g G _ H a A b B g G h _
labels   :     . . . . E . . h . . . . . . . . H
kind     :     . . . . M . . M . . . . . . . . M
           (M = [MASK]  R = random token  K = kept as-is  . = no loss)

over 10000 sentences (160000 real tokens):
  [MASK]   12.0%   (recipe says 12.0%)
  random    1.5%   (recipe says 1.5%)
  kept      1.5%   (recipe says 1.5%)
  picked   15.0%   (recipe says 15.0%) ✓

My toy corpus: a "sentence" is eight bracket pairs, an opener (a–h) followed by its own closer (A–H), openers chosen at random. So a closer is predictable from its left, an opener only from its right. Deliberate: half the blanks need right context.

Drop the mask and every token sees both sides

The attention page's scaled_dot_product_attention, same input, one argument changed. Row = a token; column = how much of another it reads. 

no mask (BERT) -- each row = what that token can see:
  b  [0.79 0.01 0.04 0.04 0.11]   sees 5 of 5
  B  [0.02 0.86 0.04 0.03 0.05]   sees 5 of 5
  e  [0.20 0.10 0.28 0.21 0.22]   sees 5 of 5 ← reads the E to its right
  E  [0.20 0.07 0.25 0.27 0.21]   sees 5 of 5
  a  [0.39 0.09 0.17 0.13 0.23]   sees 5 of 5

causal mask (GPT) -- each row = what that token can see:
  b  [1.00 0.00 0.00 0.00 0.00]   sees 1 of 5
  B  [0.03 0.97 0.00 0.00 0.00]   sees 2 of 5
  e  [0.34 0.17 0.49 0.00 0.00]   sees 3 of 5 ← has no idea an E is coming
  E  [0.26 0.09 0.32 0.34 0.00]   sees 4 of 5
  a  [0.39 0.09 0.17 0.13 0.23]   sees 5 of 5
Same code, one argument. mask=None is BERT; mask=causal_mask(n) is GPT. The last row is identical in both: the final token always sees everything, which is why a causal model is fine for generation. BERT wants every token to have that view.

Train it: loss falls, the blanks fill in

The encoder is two of the attention page's blocks in PyTorch, 32 wide, two heads, 27k parameters. BERT-base is twelve blocks, 768 wide, twelve heads, 110M parameters, a million steps on BooksCorpus plus Wikipedia. Same shape, smaller numbers. Six hundred steps on a laptop CPU: 

  step    1   loss 3.083   masked-token accuracy   5.3%
  step  200   loss 0.289   masked-token accuracy  89.4%
  step  400   loss 0.429   masked-token accuracy  89.6%
  step  600   loss 0.407   masked-token accuracy  89.4% ✓

  partner visible and intact :  99.9%  (4172 targets)
  partner hidden or corrupted:  18.3%  (613 targets, chance = 12.5%)

Why 89% and not 100%? The last two lines say why. Partner visible: near perfect. Both halves of a pair picked: nothing left to infer from. That 11% is the recipe, not the model.

A held-out sentence, blanks filled in by the model:

  original : [CLS] a A e E h H a A g G g G c C f F
  input    : [CLS] a A e E h H a _ g G g G c b f _
  predicted: [CLS] a A e E h H a A g G g G c C f F   ✓✓✓

Two blanks filled. And the b after c was a random replacement, not a blank; the model still corrected it. The 10% random rule.

Keep the mask and watch it fail

Now the paper's actual argument. Same model, same data stream, same seed, same 600 steps. The only change is causal=True: the attention page's lower-triangular mask, put back. 

                                  bidirectional    causal
  all masked tokens                       89.4%     55.5%
    closers (clue on the LEFT)            89.6%     89.7% ✓
    openers (clue on the RIGHT)           89.3%     20.4%
  chance = 12.5%

Look at the split. The causal model closes brackets as well as BERT does; the opener is on its left. It cannot guess an opener: the only evidence is the closer on its right, and the mask hides it. 20%, barely above random. Half of language has its clue on the right. That is the paper's entire claim, in one row.

The interview version. "Why can't you use BERT to generate text?" Because it fills blanks using the right context, and at generation time there is no right context yet. And it was never trained to predict the next token, only hidden ones. That job went to the GPT family, which keeps the mask and pays the 20%.

[CLS], NSP, and a different main() per task

BERT prepends one extra token, [CLS], to every input. No meaning; a slot. Its output at the top of the stack is "the sentence": encoder output (batch, 17, 32), row 0 is (batch, 32), one linear layer gives (batch, 2). That's the whole classification head.

The paper's second pre-training task, next sentence prediction, gives that slot a job. Take sentences A and B; half the time B really followed A, half the time it's random. Feed [CLS] A [SEP] B [SEP] plus a learned segment embedding, and train the [CLS] output to say IsNext or NotNext. The ablation in §5.1 says dropping it hurts QNLI, MNLI and SQuAD. RoBERTa, a year later, trained without NSP on more data and matched or beat BERT, so the field dropped it. I didn't implement it, and you can see the cost: [CLS] has done nothing in pre-training.

Fine-tuning fixes that, but only if the gradients reach the encoder. Toy task, a miniature CoLA: is this sentence grammatical? Half have one closer swapped for a wrong one. 

  300 steps of fine-tuning        trainable params  accuracy
  head only, encoder frozen                     66     51.8%
  everything (the paper)                     27188     99.2%
  everything, NO pre-training                27188     50.1%
  chance = 50.0%

Three rows, both of the paper's claims. Freeze the encoder: chance. Fine-tune everything: solved in 300 steps. Fine-tune everything from a random encoder for the same 300 steps: chance again. Pre-training makes fine-tuning cheap; fine-tuning all of it makes pre-training usable. The paper does both, on all 110M parameters, at 2e-5 to 5e-5 for two to four epochs. In my toy the frozen head gets nothing; in the paper's own feature-based experiment on NER it gets most of the way, because twelve layers of real features with NSP behind them beat my 32 numbers.

The code

NumPy for the mechanism: the masking recipe and the attention masks. A tiny PyTorch model, CPU only, trains in under 30 seconds for the one part that needs gradients. Every function under 20 lines. The new parts are below; the NumPy attention is the attention page's, copied.

Download the full script — about 200 lines of mechanism plus six runnable demos. Then:

python3 bert_from_scratch.py
def mask_batch(ids, rng, p_pick=0.15, p_mask=0.8, p_random=0.1):
    corrupted, labels = ids.copy(), np.full_like(ids, -1)
    kind = np.zeros_like(ids)          # 0 untouched, 1 [MASK], 2 random, 3 kept
    picked = (rng.random(ids.shape) < p_pick) & (ids != CLS)   # never pick [CLS]
    labels[picked] = ids[picked]           # the answer key
    roll = rng.random(ids.shape)
    to_mask = picked & (roll < p_mask)
    to_random = picked & (roll >= p_mask) & (roll < p_mask + p_random)
    to_keep = picked & ~to_mask & ~to_random
    corrupted[to_mask] = MASK
    corrupted[to_random] = rng.integers(OPEN0, VOCAB, int(to_random.sum()))
    kind[to_mask], kind[to_random], kind[to_keep] = 1, 2, 3
    return corrupted, labels, kind


class TinyBERT(nn.Module):
    def __init__(self, causal=False, d=32, heads=2, layers=2):
        super().__init__()
        self.tok = nn.Embedding(VOCAB, d)
        self.pos = nn.Embedding(L, d)         # BERT learns positions, not sines
        self.blocks = nn.ModuleList(Block(d, heads) for _ in range(layers))
        self.mlm_head = nn.Linear(d, VOCAB)   # "which token was here?"
        # THE SWITCH. None = every token sees both sides (BERT).
        # -inf above the diagonal = every token sees only its left (GPT).
        self.attn_mask = (torch.triu(torch.full((L, L), float("-inf")), 1)
                          if causal else None)

    def encode(self, ids):                    # (batch, L) -> (batch, L, d)
        x = self.tok(ids) + self.pos(torch.arange(L))
        for block in self.blocks:
            x = block(x, self.attn_mask)
        return x

    def forward(self, ids):                   # (batch, L) -> (batch, L, VOCAB)
        return self.mlm_head(self.encode(ids))


def mlm_loss(logits, labels):
    labels = torch.from_numpy(labels)
    if int((labels >= 0).sum()) == 0:         # nothing picked -> nothing to learn
        return logits.sum() * 0.0
    return F.cross_entropy(logits.reshape(-1, VOCAB), labels.reshape(-1),
                           ignore_index=-1)

Now break it

This is the part that actually makes it stick. Reading won't do it. Every row here is one edit and one number the script will print back.

Break thisWatch for
In demo_3_train_bert, change TinyBERT(causal=False) to causal=TrueMasked-token accuracy drops from 89.4% to 55.5%. Openers, whose only clue is on the right, fall to 20.4%. Right context is invisible.
Change the mask_batch defaults to p_mask=1.0, p_random=0.0Accuracy at kept-as-is positions goes from 99.3% to 53.0%. The model has never been marked on a visible token; fine-tuning will show it nothing else.
Change the default to p_pick=0.5On its own 50%-masked data it scores 64.2%: only 55% of blanks still have their partner visible. The rule is learned; the task became unanswerable.
Change the default to p_pick=0.0Loss prints 0.000 from step 1 and never moves. No blanks, no loss, no gradient. Accuracy 8.3%, below chance.
Freeze the encoder: p.requires_grad_(False) on every encoder parameter66 trainable parameters, 51.8% on the grammar task. Chance. The paper fine-tunes all 110M for a reason.
Skip pre-training: Classifier(TinyBERT()) instead of Classifier(copy.deepcopy(bert))Same 300 steps, all 27188 parameters trainable, 50.1%. Fine-tuning a random encoder is training from scratch, slowly.
Then read the paper — but only §3, the method, and §5.1, the ablation that removes NSP and then bidirectionality. Skip the GLUE and SQuAD tables in §4; every number in them has been beaten since. You'll find the paper is just the comments you've already read.

On the maths anxiety

New maths on this page, in full: cross-entropy is −log of the probability the model put on the right answer, averaged over the blanks. A log so a small probability becomes a big penalty. That's it. Everything else is the attention page's maths, and you cleared that.

What actually bites in practice is not maths: WordPiece splitting a word you care about into three pieces, the 512-token limit, which layer to take features from, and whether three epochs of fine-tuning silently overfit. Those are engineering, and they're what an interview will actually probe.

Where to go next

This page gave you one thing: BERT as a cloze test, with a model you can break. That's deliberately narrow. These cover what I skipped.

Order I'd take them in: Illustrated BERT → the paper's §3 and §5.1 → the first-time guide. Pictures, source, then hands on a real model.