GPT: Generative Pre-trained Transformers (GPT-1 to GPT-3)

Radford et al., 2018 · Radford et al., 2019 · Brown et al., 2020

Three papers, one while loop. Explained as a program, because the program is the whole idea.

Download gpt_from_scratch.py — everything below runs from it. NumPy, plus a tiny CPU PyTorch model for the training step.

I'm an AI Engineer. I've shipped models, done the MLOps, and I've called the GPT API more times than I've called my mother. I still couldn't have told you what the three papers actually claim, or why one title says "unsupervised multitask" and the next says "few-shot". Nobody told me they are one idea with three headlines, and that the idea is a loop I've written a hundred times.

Autocomplete that got so good it became an interface

You know this:

text = "the model "
while len(text) < 100:
    text += most_likely_next_char(text)    # feed the output back in

That's GPT. All three of them. The only thing that changed between 2018 and 2020 is what lives inside most_likely_next_char — a stack of decoder blocks from the attention page, causal mask on — and how many of them: 

def most_likely_next_char(text):
    ctx = encode(text)[-block_size:]      # crop to the context window
    logits, _ = gpt_forward(ctx, p)       # (T, vocab): a guess for EVERY row
    return decode([argmax(logits[-1])])   # keep only the last row

Type a question and it types the answer, because the answer is the likeliest continuation of a question. The model doesn't know it's answering. It's finishing your sentence.

TermMeansProgrammer's analogy
TokenThe unit of text. A character in my script, a word-piece in the real thingThe key type of the dict
Context windowThe most tokens the model can see at once (block_size)A ring buffer. Old tokens fall off the back
Causal maskNo position may read a later positionA read-lock on the future
TemperatureDivide the logits before softmaxA knob between argmax and dice
In-context learningWorked examples in the prompt, weights frozenConfiguration by input, not by redeploy

One idea, three headlines

Before GPT-1 you built one architecture per task and paid for a labelled dataset for each. GPT-1's pitch: the labels are free if the task is "predict the next token". GPT-2 dropped the fine-tuning. GPT-3 put the examples in the prompt. Same block, same loss, same loop, more of everything. Two details you'll meet in any GPT codebase: position embeddings are learned, not sine waves, and from GPT-2 on the LayerNorm sits at the input of each sub-layer. Neither is the idea.

PaperSizeDataThe headline claim
GPT-1, 201812 layers, dmodel 768, 117M paramsBooksCorpus, ~7,000 unpublished booksPretrain on next-token, then fine-tune per task by rewriting the task's inputs as one token sequence with delimiters. Pretraining transfers.
GPT-2, 201948 layers, dmodel 1600, 1.5B paramsWebText, ~40 GB of pages linked from RedditNo fine-tuning at all. Phrase the task as text ("TL;DR:") and read the continuation. Zero-shot from scale.
GPT-3, 202096 layers, dmodel 12288, 96 heads, 175B params~300B tokens: filtered Common Crawl, WebText2, books, WikipediaPut a few worked examples in the prompt and the frozen model uses them. In-context learning, no gradient updates.

The one formula

L(θ) = −Σt log pθ(xt | xt−k, …, xt−1)

GPT-1's Equation 1 with the sign flipped, so it's a loss to minimise rather than a likelihood to maximise. k is the context window. Everything else in three papers is this sum, made bigger. Read it as pipeline stages:

StepShapeWhat it does
tokens(T,)Integers. The text is its own label — nobody annotated anything
tok_emb[tokens] + pos_emb[:T](T, d)What each token is, plus where it sits. Both learned
decoder_block × N(T, d)Causal attention (tokens talk, backwards only), then an MLP (each token thinks alone). Shape unchanged → stackable
layer_norm @ W_out(T, V)Logits. One guess for every position, all at once — one matrix multiply scores T rows. The mask makes that legal: row t is scored on tokens[t+1] and never saw it
softmax, row-wise(T, V)Percentages
−log p[target](T,)How surprised each row was by the real next token. 0 if certain and right
meanscalarThe loss. ln(V) for a uniform guess; the floor is the data's own randomness

The loop: feed the output back in

Demo 1 runs the NumPy block with random weights. Nothing is learned; I just want you to see the shapes. Four tokens in, a (4, 28) matrix out — one guess per position — and the loop keeps only the last row:

Don't take my word for it. Here's the actual output:

logits                        : (4, 28)  (one row PER position)
we keep only logits[-1]       : (28,)  (the next-token guess)

the loop, 5 steps, greedy:
  step 0: T=4 tokens -> logits(4, 28) -> argmax of last row = 23 = 't'
  step 1: T=5 tokens -> logits(5, 28) -> argmax of last row = 16 = 'l'
  step 2: T=6 tokens -> logits(6, 28) -> argmax of last row =  0 = '\n'
  step 3: T=7 tokens -> logits(7, 28) -> argmax of last row = 23 = 't'
  step 4: T=8 tokens -> logits(8, 28) -> argmax of last row = 23 = 't'
result: 'the tl\ntt'

causal: logits for position 0 identical at ctx=4 and ctx=8? True
no mask: logits for position 0 identical at ctx=4 and ctx=8? False

The last two lines are a bonus the mask gives you for free. With it on, row 0's output never changes as the sequence grows, so a server can cache every earlier row instead of recomputing it. That's the KV cache. Without it, nothing can be cached.

The context window

The loop has no memory past block_size; GPT-3's is 2048 tokens. Demo 5 gets it wrong both ways, on the trained model: 

torch: 33 tokens into the block_size=32 trained model, no crop:
  IndexError: index out of range in self   (pos_emb has only 32 rows)

crop from the wrong end -- idx[:, :block_size] instead of idx[:, -block_size:]:
  the model hides the answer again......................................
  every token after the window filled is the same one? True
  vs the right crop, same model, same prompt:
  the model hides the answer again. the prompt hides the answer again. t

Forget to crop and the crash is the good outcome. Crop from the head and the model re-reads the same stale prefix forever, never sees what it just wrote, and prints one character until you kill it.

The shift: the label is the next character

Inputs are tokens[:-1], targets are tokens[1:]. That one-token shift is the entire labelling budget of GPT-3: 

text     : autocomplete
inputs   : autocomplet   <- tokens[:-1]
targets  :  utocomplete   <- tokens[1:], shifted one left
pairs    : a->u u->t t->o o->c c->o o->m m->p p->l l->e e->t t->e

Now break it. Set targets = inputs and train 200 steps next to the honest version:

honest  (targets = tokens[1:]), 200 steps:
  step     1  loss 3.536
  step    50  loss 1.446
  step   200  loss 0.313
BROKEN  (targets = inputs)  , 200 steps:
  step     1  loss 3.719
  step    50  loss 0.016
  step   200  loss 0.002  ← learned output[i] = input[i], nothing else
Loss near zero is not success. It's a smell. The broken model found the identity function in fifty steps and stopped. Check the shift first, every time a language-model loss looks too good.

Train it, then let it talk

Demo 3 trains a two-layer, 105,728-parameter GPT on my laptop's CPU in about ten seconds. The corpus is 400 sentences I generate from four word lists — subject, verb, object, adjunct, each drawn uniformly — so the model has a grammar to learn and can't memorise. That also means I know the floor: four choices of six per sentence is 4·ln 6 nats over about 38 characters, roughly 0.19 per character. 

  step     1  loss 3.536
  step   150  loss 0.363
  step   300  loss 0.288
  step   600  loss 0.305
  a uniform guess scores ln(28) = 3.332; the corpus's own
  randomness puts the floor near 0.191 (4 choices of 6 per sentence)

temperature 0.8, 100 chars:
  the model feeds the future by hand. the model feeds the next token again
  . the prompt feeds the answer at scale

Same loop as demo 1. The random weights printed 'the tl\ntt'; the trained weights print sentences in the corpus's grammar. Nothing in the loop changed. It learned the structure, not the meaning — there is no meaning in my corpus to learn, and that's the honest version of what a language model does.

Temperature

One number between the logits and the dice. Divide the last row by it before softmax: below 1 sharpens toward argmax, above 1 flattens toward uniform. I print the entropy of the distribution each character was drawn from, in nats; ln 28 ≈ 3.33 means "no idea": 

temperature 0.2:  mean entropy 0.096 nats
  feeds the answer again. the mask feeds the answer at scale. the mask cop
  ies the next token at scale.

temperature 2.0:  mean entropy 1.100 nats
  feeds its own oly.\nthefix by hagan. tby\naby hok fedsy . the feeds k\nt
  he prefix slowly.\nfeefcashi rec

temperature 0 for 400 chars -- does it loop?
  yes: the tail repeats every 35 chars: 'r again. the prompt hides the answe'

That last line is the "repetition" bug everyone hits. Greedy decoding is deterministic, so the moment it revisits a state it's in a cycle forever. Temperature 0 isn't "the most accurate setting"; it's a loop with no exit condition.

The mask is what makes the loop honest

The attention page said: remove the mask and you have BERT. Here's what that costs if you wanted to generate. Demo 6 trains the identical model with causal=False, same data, same 600 steps: 

same model, same data, same steps, mask removed:
  step     1  loss 3.540
  step   600  loss 0.013  ← vs 0.305 with the mask on

loss per position: mean over positions 0..30, then position 31,
the one row that has no next column to peek at:
  causal : 0.280   last position: 0.190
  no mask: 0.001   last position: 0.296

short prompt 'the model ', temperature 0.8:
  causal :
    the model feeds the future by hand. the model feeds the next token again
    . the prompt feeds the answer at scale
  no mask:
    the model ooooooomoooooooooooooooooooooooooooooooooooooooooooooooooooooo
    oooooooooooooooooooooooooooooooooooooo

Without the mask, position i can attend to position i+1 — which is its target. So it copies, and the loss collapses to 0.001 at every row but the last, the only row with nothing to copy. At sampling time the last row is the only row that matters, so 31 of 32 rows of training went on a trick that never fires in production, and the one honest row got 1/32 of the gradient: 0.296 against the masked model's 0.190. Give it a full 32-character window so the last row is the honest one and it limps — run-on sentences, a made-up word; the script prints that too. Train/serve skew, controlled by one flag.

Lower training loss, worse model. The mask isn't there to make training easier. It's there to make training the same problem as serving. Bidirectional attention is BERT, and BERT doesn't generate.

In-context learning: the prompt is the program

GPT-3's headline. Say it plainly, because interviewers will ask: in-context learning means the worked examples are in the prompt and the weights are frozen. No fine-tuning, no gradient step, nothing written to disk. The paper's few-shot format is exactly this shape — a few input→output pairs, then an input with no output: 

english: cat -> french: chat
english: dog -> french: chien
english: house -> french: maison
english: cheese -> french:

that prompt is 118 tokens; the model sees one integer array:
  [10, 18, 12, 16, 14, 22, 13, 4, 1, 8, 6, 23] ... [1, 11, 21, 10, 18, 8, 13, 4]
logits shape: (1, 32, 28)   gradient updates: 0

Nothing in the code knows what an "example" is. The three worked pairs are just more prefix; few-shot is longer input, full stop. GPT-3's claim is that at 175B parameters the likeliest continuation is fromage, and that this improves with more examples and more parameters — the results section is that curve, task by task. My model has 105,728 parameters and a six-noun vocabulary. It won't do this, and I'm not going to pretend it does. The mechanism is identical. The capability is what scale bought. What came after — making the autocomplete want to follow instructions rather than merely continue them — is InstructGPT.

The code

NumPy for the mechanism. A tiny PyTorch model, CPU only, trains in about ten seconds for the one part that needs gradients. Every function under 16 lines. Attention, the multi-head wrapper and the causal mask are the attention page's, with a causal switch added so a demo can cheat.

Download the full script — 130 lines of mechanism plus seven runnable demos. Then: 

python3 gpt_from_scratch.py
def decoder_block(x, p, causal=True):
    a, weights = multi_head_attention(layer_norm(x), p, causal)
    x = x + a                                        # tokens talk (backwards only)
    x = x + gelu(layer_norm(x) @ p["W_1"]) @ p["W_2"]  # each token thinks alone
    return x, weights                                # shape unchanged -> stackable


def gpt_forward(tokens, p, causal=True):
    T = len(tokens)
    assert T <= p["block_size"], f"context {T} > block_size {p['block_size']}"
    x = p["tok_emb"][tokens] + p["pos_emb"][:T]      # (T, d): what + where
    x, weights = decoder_block(x, p, causal)         # a real GPT stacks 12..96
    logits = layer_norm(x) @ p["W_out"]              # (T, vocab): a guess PER row
    return logits, weights


def generate(p, prefix, n_new, temperature=0.0, rng=None, trace=False):
    tokens = list(prefix)
    for step in range(n_new):
        ctx = tokens[-p["block_size"]:]              # crop: no memory past this
        logits, _ = gpt_forward(np.array(ctx), p)
        row = logits[-1]                             # only the LAST row matters
        if temperature == 0:
            nxt = int(np.argmax(row))                # greedy
        else:
            nxt = int(rng.choice(len(row), p=softmax(row / temperature)))
        tokens.append(nxt)                           # feed it back in
        if trace:
            print(f"  step {step}: T={len(ctx)} tokens -> logits{logits.shape}"
                  f" -> argmax of last row = {nxt:2d} = {decode([nxt])!r}")
    return tokens


def get_batch(data, block_size, batch_size, gen, shift=True):
    ix = torch.randint(len(data) - block_size - 1, (batch_size,), generator=gen)
    x = torch.stack([data[i:i + block_size] for i in ix])
    if shift:
        y = torch.stack([data[i + 1:i + block_size + 1] for i in ix])  # THE shift
    else:
        y = x.clone()                                # the break: label == input
    return x, y

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living — every row below is one edit to the script and one thing to watch.

Break thisWatch for
In demo_3_train_and_sample, build the model with causal=FalseTraining loss 0.013 instead of 0.305 — it's reading the answer off the next column. Sample from a short prompt: ooooooo… forever.
In get_batch, replace the shifted y with y = x.clone()Loss 0.002 by step 200. It learned output[i] = input[i] and nothing about the text.
Sample with temperature=0.0 for 400 charactersThe tail repeats every 35 characters — "the prompt hides the answer again." — until you stop it. Deterministic loop, no exit.
In sample, delete the crop: model(idx) instead of model(idx[:, -model.block_size:])IndexError: index out of range in self on the 33rd token. pos_emb has 32 rows. The crash is the good outcome.
Crop from the head instead: idx[:, :model.block_size]Re-reads the same stale prefix every step and prints . forever once the window fills. No crash, just wrong.
In demo_1_the_loop, call gpt_forward(..., causal=False) and print weights[0]The upper triangle fills in; row 0 now reads row 3, and position 0's logits change every time you append a token, so nothing can be cached. That's an encoder block. It's BERT.
Then read the papers — but only these bits. GPT-1: §3.1 for the loss (Equation 1) and §3.3 with Figure 1 for how a task becomes one token sequence. GPT-2: §2, especially the argument that a task is just a conditioning prefix. GPT-3: §2 and Figure 2.1, Table 2.1 for the sizes, then §5 Limitations. Skip every benchmark table on a first pass. You'll find the papers are the comments you've already read.

On the maths anxiety

Honestly: the whole thing is one −log. Cross-entropy is minus the log of the probability the model gave the right answer — 0 when it was sure and right, big when it was sure and wrong. Softmax turns scores into percentages. Temperature divides before the softmax. Entropy is "how surprised do I expect to be", the same −log averaged. That's every symbol on this page.

You don't need to derive anything; loss.backward() and F.cross_entropy are library calls. What I'd want you to do at a whiteboard is write the shift, write the mask, and say why removing either one makes the loss go down and the model go bad. What separates AI engineers here isn't research maths: it's knowing that in-context learning is a longer input, that the KV cache exists because of the mask, that context length is a hard ring buffer, and that temperature 0 loops.

Where to go next

This page gave you one thing: three papers as one loop you can break. It skipped byte-pair tokenisation, the fine-tuning recipe, top-k and nucleus sampling, and the entire results section. These cover what I cut.

Order I'd take them in: Alammar → Karpathy → the paper. Pictures, then the same code done properly, then the claims. If you only take one, take Karpathy — it's this page with a working GPU and a better narrator.