InstructGPT: Training Language Models to Follow Instructions with Human Feedback

Ouyang et al., 2022

The recipe that turned a text completer into an assistant, run end to end on a toy small enough to break.

Download instructgpt_from_scratch.py — everything below runs from it. NumPy for the world, a few hundred steps of CPU PyTorch for the three training loops. About two seconds.

I'd used GPT-3 through the API, and I'd used ChatGPT, and I could not have told you what happened in between. This paper is the in-between. I expected the hard part to be the RL. It isn't — the RL is one line. The hard part was seeing that there are three separate problems, and each stage fixes one.

Onboarding a new hire

A new engineer joins. Smart, has read everything, and on day one still doesn't know how you do things. So you show them a few worked examples. Then you review their work — "this one, not that one" — until they pick up your taste. Then they practise on real tickets, with a mentor nearby to say "you're drifting, come back" the moment they refactor the codebase to fix a typo. That's the paper. Before it, GPT-3 had one job: 

prompt = "Explain what a mutex is."
gpt3(prompt)   # "what comes next in a document containing this line?"
               # -> "Explain what a semaphore is. Explain what a deadlock is."

Not stupid, not stubborn. A document containing "Explain what a mutex is." is usually an exam paper, and the next line of an exam paper is the next question. Every trick for making GPT-3 behave — few-shot examples, "Q: … A:" scaffolding — was a way of forging a document in which the answer happens to come next. InstructGPT is three stages on top of that: 

policy = base                                    # completes text, does not answer
policy = fit(policy, demonstrations)             # 1. SFT: copy the worked examples
reward = fit(comparisons(policy.samples()))      # 2. RM: learn their taste
policy = argmax(reward(policy) - beta * KL(policy, sft))   # 3. RL, on a leash
TermMeansNew-hire analogy
DemonstrationA human wrote the answer they wantedThe worked example on day one
ComparisonA human picked the better of two model outputsCode review: "this one, not that one"
Reward modelA network trained to predict which output a human prefersThe reviewer's taste, compiled
KL penaltyA cost for drifting away from the SFT modelThe mentor saying "come back"

The headline: on real prompts people had sent to the API, labelers preferred the 1.3B InstructGPT to the 175B GPT-3, a model a hundred times bigger. The 175B InstructGPT beat 175B GPT-3 85% of the time, and 71% of the time against GPT-3 with a carefully written few-shot prompt. That's the interview answer to "why doesn't scaling fix this?": you cannot scale your way out of the wrong objective.

The one formula

objective(φ) = E(x,y)∼πφ[ rθ(x, y) − β log(πφ(y|x) / πSFT(y|x)) ] + γ Ex∼Dpretrain[ log πφ(x) ]

That's the stage-3 objective from the method section. Three things being traded off:

TermShapeWhat it does
y ∼ πφsamplesThe policy answers its own prompts. PPO estimates the expectation by sampling; my toy has six candidates per prompt, so I compute it exactly
rθ(x, y)scalarThe reward model's score. "What a labeler would think", compiled into a network
β log(πφ / πSFT)scalar, per tokenThe leash. Averaged over samples it's KL(πφ ‖ πSFT): how far the policy has drifted from stage 1. β = 0.02 in the paper
γ E[log πφ(x)]pretraining lossPPO-ptx. Keep doing the original job on pretraining data while you practise. γ = 27.8 in the paper; γ = 0 is plain PPO

Drop the γ term and the objective has an exact answer: π*(y|x) ∝ πSFT(y|x) · er(x,y)/β — the SFT distribution, re-weighted by the reward. β → ∞ is SFT; β → 0 is the reward model's argmax. The script checks the trained policy lands there (max difference 0.0005).

Stage 0: GPT-3 completes; it does not answer

The toy world: four prompts, six candidate responses each, and a hidden helpfulness score (0 to 10) that nothing in training ever reads — it only grades the pipeline afterwards. A policy is a 4×6 table of logits; answering means softmax the row and sample. Per prompt: three continuations, a terse answer, a good answer, a padded "Great question!" essay. The base logits make a continuation e2 ≈ 7× more natural than a one-word answer, and an answer rarer the longer it gets: 

  prompt:    "Explain what a mutex is."
  base says: "Explain what a semaphore is. Explain what a deadlock is."

  base             mass on answers: 0.053   true helpfulness: 0.79
  base policy on prompt 0: continuations 0.965, terse 0.030,
                           good 0.0044, padded 0.0005

Stage 1: SFT teaches the format

Labelers write the answer they want to see — about 13k prompts' worth. Then plain supervised fine-tuning: cross-entropy on those answers, the same loss as pretraining, 16 epochs. Nothing new in the loss; what's new is the data. Here it's eight demonstrations: per prompt, the labeler wrote the terse answer once and the good answer once. 

  base             mass on answers: 0.053   true helpfulness: 0.79
  SFT              mass on answers: 0.876   true helpfulness: 5.69

  SFT policy on prompt 0, per response: [0.043 0.043 0.043 0.442 0.428 0.001]
  (terse answer 0.44, good answer 0.43, padded answer 0.0006)   ← a coin flip

Eight examples took it from 5% answering to 88%. That is what SFT is for: the format. But look at the split. SFT copies what it was shown, and it was shown a terse answer and a good one equally often, so it has no opinion about which is better. To give it one you'd have to write down what "better" means. You can't.

Stage 2: a reward model captures what you can't write down

Labelers stop writing. They see several of the model's own outputs for a prompt and rank them — 4 to 9 at a time in the paper, every pair from one ranking trained as a single batch, about 33k prompts. I can't define a good answer, but I can tell you which of two is better. Turning "a beat b" into a scalar is Bradley-Terry: P(a beats b) = sigmoid(r(a) − r(b)), minimise −log of that. The RLHF page derives it; here it's three lines. The paper's reward model is a 6B transformer with a scalar head. Mine is a linear layer over three numbers per response: does it answer, how long is it, that length squared. Same loss, same lesson. The labelers are simulated and imperfect; I tuned their noise until two of them agree about as often as the paper's real ones did, 72.6%. 

  comparisons collected on SFT outputs: 240
  simulated inter-labeler agreement:    0.737   (paper: about 0.73)
  padded answers that reached a labeler: 0 of 4

  reward model vs hidden score, on responses labelers saw: corr = 0.979
  reward model vs hidden score, on ALL 24 responses:       corr = 0.708

  prompt 0, learned reward vs hidden score:
    reward  1.96   hidden 0.5   'Explain what a semaphore is. E'
    reward  4.60   hidden 5.0   'A lock one thread holds at a t'
    reward  6.12   hidden 8.0   'A lock one thread holds while '
    reward  7.86   hidden 3.5   'Great question! A mutex, or mu'   <- never seen

The reward model is near-perfect on what labelers actually saw: 0.979. And it is guessing about what they never saw. The padded essay has 0.06% probability under SFT, so no labeler was ever shown one. The RM learned "longer is better" from terse-vs-good — true on that data — and extrapolated straight past the good answer to the essay. Hold that thought.

A reward model cannot be more consistent than the people it learned from. The script reruns this with labelers who agree only 54% of the time: the ranking survives (corr 0.973) but the good-minus-terse reward gap goes from 1.52 to 0.09, and margins are what stage 3 trades against the leash. The paper's 73% is a ceiling on the whole pipeline, which is why §3.4 spends so long on who the labelers were.

Stage 3: practise against the reward model, on a leash

Now the policy answers prompts (about 31k), the reward model scores the answers, and the policy is updated to score higher — with a per-token KL penalty against the SFT model so it can't wander off. The paper uses PPO; the RLHF page walks through the sampling and clipping and I'm not re-deriving it. With six candidates per prompt I compute the expectation exactly and let autograd do the rest. 

  prompt:    "Explain what a mutex is."
  RLHF says: "A lock one thread holds while the rest wait; guards shared state."

  stage          mass on answers   true helpfulness   KL to SFT
  base                     0.053               0.79       1.795
  sft                      0.876               5.69       0.000
  rlhf                     0.999               7.68       0.539

  ceiling (always the good answer): 8.12
  RLHF policy on prompt 0: terse 0.10, good 0.88, padded 0.015
  closed form pi_SFT * exp(r/beta): max |diff| from trained policy = 0.0005

RL did the one thing SFT couldn't: it moved the coin flip to 0.10 / 0.88, because the reward model ranks the two. Helpfulness 0.79 → 5.69 → 7.68 against a ceiling of 8.12. And the padded essay crept up to 1.5% — the RM over-rates it, the policy leaned that way, and the leash is the only reason it stopped. Two experiments show what the leash is for.

Point the leash at the wrong model

Skip SFT — start RL from the base model and leash it to the base model, because that's the only model you have. Or do SFT, then point the KL at the base model. 

  steps until true helpfulness >= 7.0, then the final numbers:

  run                     steps   mass on answers   helpfulness
  SFT -> RL, KL to SFT       7             0.999          7.68
  base -> RL, KL to base never             0.892          4.75
  SFT -> RL, KL to base  never             0.892          4.75

  helpfulness at step 10:  with SFT 7.41   without SFT 1.50

From the base model the good answer is a rounding error, so RL has almost nothing to push on and crawls: 1.50 after ten steps against 7.41. And the leash pulls back toward continuations the whole way, so it settles at 4.75 and never arrives. The two bad rows end in the same place: the reference model decides the destination, the start only decides the speed. (With real sampling, the start also decides whether you ever see a good answer at all.)

Take the leash off

  beta = 0 says: "Great question! A mutex, or mutual exclusion... [4 paragraphs]"
  probability on the padded answer, prompt 0: 0.999
  beta = 0.7       mass on answers: 0.999   true helpfulness: 7.68
  beta = 0         mass on answers: 1.000   true helpfulness: 3.50

  + 240 comparisons on the drifted policy's own outputs, RM retrained:
    corr with hidden score on ALL 24 responses: 0.979   (was 0.708)
    RM on prompt 0: good 6.28  padded 4.04   (was good 6.12, padded 7.86)
  retrained RM     mass on answers: 0.998   true helpfulness: 7.63

With β = 0 the policy runs straight to whatever the reward model over-rates: the essay, 99.9% of the time, and true helpfulness drops from 7.68 to 3.50. That's reward hacking, and the RM was never wrong on its own data — it was asked about something it had never seen. The paper has two guards: the KL penalty, and collecting comparisons on outputs from the models being trained, not only the SFT model, then retraining the RM on all of it — so it's trained where the policy actually lives. The bottom half of the block is that fix: 240 fresh comparisons on the drifted policy's outputs, the RM learns the essay is worse than the good answer (4.04 against 6.28), and the policy comes back.

ChatGPT is this recipe. The same three stages, with the demonstrations and comparisons written as dialogues instead of single turns. If you can explain these stages and what each one buys, you can explain ChatGPT.

The alignment tax, and PPO-ptx

Stage 3 made the model better at what labelers wanted and worse at things nobody was checking: after RLHF, scores dropped on public benchmarks like SQuAD, DROP, HellaSwag and a French→English translation task. The paper calls this the alignment tax. It's distribution shift again — practise on 31k API-style prompts and you drift from what the pretraining corpus taught you, even on the leash. The fix is the γ term: while running PPO, keep taking ordinary next-token gradient steps on batches of pretraining data. They call it PPO-ptx, and it recovers most of the lost benchmark performance at little cost in labeler preference. Programmer version: keep the old regression suite running while you refactor. It's not in the script because the toy has no pretraining corpus to mix in; it's one extra loss term, added, and the interesting part is why it's there.

The code

NumPy for the toy world and every measurement. PyTorch, CPU only, for the three training loops — each stage is a training objective, and hand-written backprop would bury that. No function longer than 15 lines.

Download the full script — about 230 lines including the toy world, plus six runnable demos. Then: 

python3 instructgpt_from_scratch.py
# STAGE 1 -- SFT (supervised fine-tuning). Section 3.5 of the paper.

def sft(base_logits, demos, steps=60, lr=1.0):
    """demos[p] = indices of the responses a labeler wrote for prompt p."""
    theta = torch.tensor(base_logits, requires_grad=True)
    opt = torch.optim.SGD([theta], lr=lr)
    prompts = torch.tensor([p for p, ds in enumerate(demos) for _ in ds])
    targets = torch.tensor([d for ds in demos for d in ds])
    for _ in range(steps):
        logp = torch.log_softmax(theta, dim=-1)
        loss = -logp[prompts, targets].mean()      # cross-entropy on the demos
        opt.zero_grad(); loss.backward(); opt.step()
    return theta.detach().numpy()

# STAGE 2 -- reward model from comparisons. Section 3.5 of the paper.

def labeler(p, a, b, rng):
    """One human, one pair. Returns (winner, loser). Not perfectly reliable."""
    p_a_wins = 1 / (1 + np.exp(-(HIDDEN[p, a] - HIDDEN[p, b]) / LABELER_TEMP))
    return (a, b) if rng.random() < p_a_wins else (b, a)

def collect_comparisons(sample_logits, n, rng):
    """Sample two DISTINCT responses from the given policy, ask a labeler.
    Which policy you sample from is the whole distribution-shift story."""
    probs, out = softmax(sample_logits), []
    for _ in range(n):
        p = rng.integers(N_PROMPTS)
        a, b = rng.choice(N_RESP, size=2, replace=False, p=probs[p])
        out.append((p, *labeler(p, a, b, rng)))
    return out

def train_reward_model(comparisons, steps=400, lr=0.1):
    """Linear reward on FEATURES. Returns a (prompt, response) reward table."""
    F = torch.tensor(FEATURES)
    w = torch.zeros(F.shape[-1], requires_grad=True)
    b = torch.zeros(1, requires_grad=True)
    opt = torch.optim.Adam([w, b], lr=lr)
    ps, ws, ls = (torch.tensor(c) for c in zip(*comparisons))
    for _ in range(steps):
        r = F @ w + b                              # reward of every candidate
        margin = r[ps, ws] - r[ps, ls]             # winner minus loser
        loss = -torch.nn.functional.logsigmoid(margin).mean()   # Bradley-Terry
        opt.zero_grad(); loss.backward(); opt.step()
    return (F @ w + b).detach().numpy()

# STAGE 3 -- RL against the reward model, on a KL leash. Section 3.5.

def rlhf(init_logits, ref_logits, reward, beta, steps=300, lr=0.1):
    """Returns the whole trajectory of logits, so demos can watch it move."""
    theta = torch.tensor(init_logits, requires_grad=True)
    ref_logp = torch.log_softmax(torch.tensor(ref_logits), dim=-1)
    R = torch.tensor(reward)
    opt = torch.optim.Adam([theta], lr=lr)
    history = [init_logits.copy()]
    for _ in range(steps):
        logp = torch.log_softmax(theta, dim=-1)
        p = logp.exp()
        expected_reward = (p * R).sum(-1)              # what the RM thinks
        leash = (p * (logp - ref_logp)).sum(-1)        # KL(pi || pi_ref)
        objective = (expected_reward - beta * leash).mean()
        opt.zero_grad(); (-objective).backward(); opt.step()
        history.append(theta.detach().numpy().copy())
    return np.stack(history)

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living — reverse-engineering from broken behaviour is your native mode. Every row is a one-line change; the script already prints the before-and-after.

Break thisWatch for
Skip SFT: in demo_4, call rlhf(BASE_LOGITS, BASE_LOGITS, …)RL starts from a completion engine. Helpfulness 1.50 at step 10 instead of 7.41, never reaches 7.0, settles at 4.75 instead of 7.68.
Wrong leash: rlhf(STAGES["sft"], BASE_LOGITS, …)Same 4.75 as skipping SFT. The leash pulls toward continuations; the reference model decides where you end up.
In demo_6, retrain the RM without the fresh comparisons: drop + freshThe "was" and "now" numbers become identical — 0.708 and 0.708, padded still 7.86. An RM can't learn about outputs it never sees; that's why the paper collected comparisons on the policy's own samples.
beta=0.0 in demo_4Reward hacking. The policy puts 0.999 on the "Great question!" essay and helpfulness falls to 3.50. The RLHF page has the general case.
LABELER_TEMP = 8.0Labelers agree 0.54 of the time. The good-minus-terse reward gap goes 1.52 → 0.09 and RL on that RM lands at 6.60 instead of 7.68. The paper's real labelers: about 73%.
Then read the paper — but only §3.5 for the three models, §3.4 for who the labelers were and the 72.6% figure, §4.1 for the preference results, §4.2 for the alignment tax. Skip the toxicity and bias tables and all of §5 on a first pass. You'll find the paper is just the comments you've already read.

On the maths anxiety

There are four objects in this paper and you've met all of them. Cross-entropy is −log of the probability you gave the right answer. The Bradley-Terry loss is a sigmoid of a difference. An expectation is a weighted average. KL divergence is the expected log-ratio of two distributions — "how surprised would the SFT model be by what the new policy says" — and it's zero when they're the same model.

What I skipped is PPO's clipped objective and the value function, because they're machinery: they estimate the gradient of the formula above from a handful of samples without blowing up. You don't derive them to use them, any more than you derive Adam. The RLHF page goes through them; the GPT page is what stage 0 actually is. What an interview will probe is the data: why comparisons instead of ratings, why you compare the policy's own outputs, what the KL is for. All of that is in the six demos.

Where to go next

This page gave you three stages, each fixing one problem, on a toy you can break. Two other people fill in what I skipped.

Order I'd take them in: Hugging Face → Huyen → the paper. Pictures, then the operational reality, then the source. If you only read one, read Huyen — it's closest to the questions you'll actually get asked.