Direct Preference Optimization: the reward model was hiding in the policy

Rafailov, Sharma, Mitchell, Ermon, Manning & Finn, 2023

Two services, and it turns out one was a pure function of the other. So they deleted it and inlined the call.

Download dpo_from_scratch.py — everything below runs from it. NumPy for the algebra, a tiny CPU PyTorch loop for the training. Nine seconds.

I wrote the RLHF page and came away thinking the pipeline was irreducible. Three moving parts: a reward model fitted to votes with the Bradley-Terry loss, a policy that climbs it, and a KL penalty holding the policy near the frozen reference it started from. (One sentence each; if any of those three is new, read that page first, and the InstructGPT page for the three-stage recipe they sit inside.) Then this paper showed up and deleted the first one. Not approximated it. Deleted it, and proved it was never a separate thing.

The service that was a pure function

You have shipped this refactor. Two services, one calls the other: 

score   = reward_service.score(prompt, answer)   # a whole trained model
policy  = climb(score, start=ref, leash=beta)    # RL, sampling, the lot

And then someone reads the code properly and notices that reward_service has no state of its own. Everything it returns is derivable from what the caller already has. So you delete the service and inline the expression: 

def score(prompt, answer):                       # no model. Two log-probs.
    return beta * (log_prob(policy, answer) - log_prob(ref, answer))

That is DPO in one line, and the rest of this page is why the line is true. The paper's title says it outright: your language model is secretly a reward model. Not "can be used as one". Is one.

TermMeansProgrammer's analogy
Closed formThe exact maximiser of the RLHF objective, written down rather than searched forThe analytic solution you find after profiling the loop
The inversionRearranging that form so r is on the left and π is on the rightReading the getter and realising it's computed, not stored
Z(x)The normaliser. Depends on the prompt only, and is the one expensive termThe denominator you cancel before you ever evaluate it
Implicit rewardβ · log(π / πref), read off the trained policyThe inlined expression, still callable
βSame leash length as in RLHF, now living inside the loss instead of the rewardThe one knob you still tune by hand

The paper reports DPO matching or beating PPO-based RLHF on the tasks it tested — sentiment control, summarisation, single-turn dialogue — so the claim is parity plus simplicity, not dominance. The win is operational, and it is the kind an engineer feels immediately: one training loop instead of two, no sampling from the policy while training, no reward model to serve, no PPO hyperparameters (clip range, value head, GAE, rollout batch) to get wrong. What was a distributed system is now loss.backward() on a batch of pairs.

The one formula

π*(y|x) = πref(y|x) · exp( r(x,y) / β ) / Z(x)
⇒ r(x,y) = β · log( π*(y|x) / πref(y|x) ) + β · log Z(x)
lossDPO = −log σ( β·log(π(yw)/πref(yw)) − β·log(π(yl)/πref(yl)) )

Three lines, and only the first is new. Read them as pipeline stages:

StepShapeWhat it does
exp(r / β)(n answers,)Tilt the reference towards high reward. Big β = gentle tilt
· πref / Z(x)sums to 1Renormalise. This is the exact optimum of E[r] − β·KL — no RL needed to reach it
β·log(π*/πref)scalar per answerTake logs, move r to the left. The reward, recovered from the policy
+ β·log Z(x)scalar per promptThe leftover. Depends on x, not y — so it cancels in any comparison
−log σ(Δ)scalarBradley-Terry, unchanged from the RLHF page. Only what's inside σ changed

The last line is a binary classifier on pairs of responses. That's it. That is the whole training algorithm.

Step 1: the objective already had an answer

RLHF maximises Ey∼π[r(x,y)] − β·KL(π ‖ πref). PPO searches for the maximiser. But for a fixed prompt that objective is a concave problem over a probability distribution, and its maximiser can be written down: the reference, tilted by the exponential of the scaled reward, renormalised.

Don't take that on faith. The script climbs the same objective by brute force — 4000 steps of Adam on the logits, no formula anywhere — and compares where it lands with where the formula says it should. The toy world is the RLHF page's, unchanged: 4 prompts, 8 answers each, a hidden true reward, β = 0.5.

Don't take my word for it. Here's the actual output:

prompt 0, in probabilities:
  formula : [0.001 0.129 0.16  0.709 0.    0.    0.    0.   ]
  solver  : [0.001 0.129 0.16  0.709 0.    0.    0.    0.   ]

  max |formula - solver| over all 32 entries: 0.000018 ✓
  objective at the formula's policy: 2.319595
  objective at the solver's policy:  2.319593

Four thousand gradient steps land where one line of algebra already was. So the RL in RLHF is a numerical search for a point you could have written down — if you had r. You don't have r. That's the next step.

Step 2: invert it, and the reward falls out of the policy

Take logs of that closed form and solve for r, and r comes out as a function of the policy. The awkward leftover is β·log Z(x): Z(x) sums over every possible answer to the prompt, which for a language model is every string that exists. Nobody can compute it. Nobody has to — it depends on the prompt, not the answer, and a preference model only ever compares two answers to the same prompt. Subtract, and it's gone. 

prompt 0:
  hidden true reward : [ 0.112  2.705  2.812  3.556 -0.035 -1.623  0.899  0.67 ]
  beta * log-ratio   : [-2.82  -0.227 -0.12   0.624 -2.967 -4.555 -2.033 -2.262]
  difference         : [2.932 2.932 2.932 2.932 2.932 2.932 2.932 2.932]

the difference is CONSTANT within a prompt. That constant is
beta*log Z(x), one per prompt, and here it is:
  prompt 0:  beta*log Z =  2.9317   spread within prompt = 4.44e-16
  prompt 1:  beta*log Z =  2.4033   spread within prompt = 4.44e-16
  prompt 2:  beta*log Z =  2.8227   spread within prompt = 8.88e-16
  prompt 3:  beta*log Z =  1.1207   spread within prompt = 4.44e-16

rank correlation within each prompt, averaged: 1.000 ✓
rank correlation across all 32 answers at once:  0.849

pairwise, the constant cancels. Prompt 0, answers 3 and 7:
  true r[3] - true r[7]      =  2.8866
  recovered[3] - recovered[7] =  2.8866

Within a prompt the recovered reward ranks answers exactly as the hidden true reward does — correlation 1.000, residual spread 4·10−16, which is float noise. Across all 32 at once it drops to 0.849, and that gap is the point rather than a flaw: each prompt carries its own β·log Z offset, and comparing answers across prompts is the one thing preference data never does.

The reward model was a pure function of the policy all along. Anything you can express as a per-prompt-normalised score, you can express as a log-ratio between two policies. Fitting a separate network to do it was redundant work.

Step 3: substitute, and the second service disappears

Now put the inverted reward into the Bradley-Terry loss from the RLHF page. Both β·log Z terms cancel, r vanishes, and what's left mentions only the policy and the reference. Ten lines of PyTorch, no sampling, no rollouts, no second model to train and serve: 

margin = beta * (logp[p, w] - ref[p, w]) - beta * (logp[p, l] - ref[p, l])
loss   = -F.logsigmoid(margin).mean()

The script runs it on 2000 noisy votes and, on the same votes and the same β, runs the previous page's full RLHF pipeline — fit a reward model, REINFORCE against it with a KL penalty — so the two are comparable rather than merely described. true reward is the scoreboard neither ever sees. 

  step    1   loss 0.6931
  step  100   loss 0.4590
  step  200   loss 0.4544
  step  300   loss 0.4541
  step  400   loss 0.4541
  step  500   loss 0.4541
  step  600   loss 0.4541

  policy                      true reward    KL to ref   parts
  reference (SFT)                  1.117        0.000   1
  RLHF: reward model + RL          2.535        0.709   3
  DPO: one classifier loss         2.898        1.183   2
  best possible                    3.196          -    -

The loss floors at 0.4541 rather than 0 because the votes are noisy — a close call is nearly a coin flip and no model should fit that. Don't read much into DPO's 2.898 beating RLHF's 2.535: four prompts, eight answers, and DPO also drifted further (KL 1.183 against 0.709). What the table earns is the last column. Comparable quality, one fewer model, no sampling loop.

DPO still needs the reference model in memory. Look at the loss: log πref appears twice. It's frozen and never gets gradients, so no optimiser state, but it is a second full set of weights resident during training. "No reward model" is not "one model".

The reward model was trained. It just isn't an object

The part that made it click for me: take the DPO-trained policy, compute β·log(π/πref) — the same expression, on a policy never told what a reward is — and check it against the hidden truth. 

  ans      true    implicit
    0     0.112      -2.601
    1     2.705      -0.263
    2     2.812       0.011
    3     3.556       0.595
    4    -0.035      -2.802
    5    -1.623      -4.015
    6     0.899      -1.508
    7     0.670      -1.023

rank correlation with the hidden true reward, all 32 answers: 0.763

Nobody built a scorer, nobody scored an answer, and there is a scorer in there anyway, ordering answers roughly as people do. That's how you do best-of-n at eval time after a DPO run with no reward model deployed. The interview question is "where did the reward model go?" and the answer is "into the log-ratio".

β is the same leash, in a new place

On the RLHF page β was a coefficient on a KL term added to the reward. Here it multiplies the margin inside the loss. Same job — how far the policy is allowed to move from the reference — and the same trade. The script sweeps it twice: once on 2000 votes, once on 150. 

  beta    KL to ref   true (2000)   KL to ref   true (150 votes)
  0.05       1.946         3.191       1.901            3.115
  0.10       1.954         3.193       2.039            3.030
  0.50       1.183         2.898       2.097            2.845
  1.00       0.538         2.405       1.624             2.710
  5.00       0.035         1.460       0.151             1.659

  reference itself:  KL     0.000   true       1.117
  best possible:                     true       3.196

β = 5 barely leaves the reference either way — short leash, safe, useless. Small β means the policy trusts the votes completely and collapses onto their argmax; with 2000 votes that argmax is mostly right, so it looks free. It isn't free, it's a bet on your data. Same trade the KL coefficient made: how much do you trust what you're optimising against?

The limitation people actually argue about

The derivation assumes the pairs came from the reference policy. That assumption does real work and is easy to violate — your preference dataset was probably collected from some other model, months ago, by someone else. The script breaks it: same 2000 votes, but sampled from a long-winded policy the reference would essentially never produce. 

average reference probability of an answer in those pairs: 0.0249
same for the on-policy pairs:                              0.1805

  preference pairs drawn from    true reward    KL to ref
  the reference (as derived)          2.898        1.183
  a far-off policy                    1.927        1.241
  no training at all (reference)      1.117        0.000

Better than not training; considerably worse than on-policy pairs, at the same KL cost. DPO only gets gradient on the pairs you show it: shown long answers only, it raises the best of a bad bunch while the good short answers lose mass by normalisation alone. Nothing in the loss warns you. PPO samples from the current policy every step, so it is on-policy by construction and doesn't have this failure — it has different ones, as the RLHF page shows.

A trade, not a replacement. DPO is simpler, cheaper and more stable to train. On-policy methods still win in some settings, and iterated/online DPO variants exist precisely to close this gap. If someone says the question is settled, they're picking a side of a live argument.

The code

NumPy for the mechanism: the world, the closed form, the inversion, the KL. A torch loop for the training, because that is what DPO is — an objective, and hand-rolling its backprop would bury the idea. CPU, nine seconds. No function over 25 lines. Below, the four functions that are the paper; the toy world, the votes and the RLHF baseline are in the script.

Download the full script — 514 lines including seven runnable demos. Then: 

python3 dpo_from_scratch.py
def optimal_policy(r, ref_logp, beta):
    """The closed form above, in log space so nothing overflows. (4, 8)."""
    return log_softmax(ref_logp + r / beta)


def implicit_reward(logp, ref_logp, beta):
    """beta * log-ratio. The reward, recovered from the policy, up to a
    per-prompt constant. This is the whole 'secretly a reward model' claim."""
    return beta * (logp - ref_logp)


def dpo_loss(logp, ref_logp, p, w, l, beta, drop_reference=False,
             no_pair=False):
    """logp: torch (4, 8) log-probs of the policy. p/w/l: prompt, winner,
    loser index tensors. The two flags are the deliberate breaks."""
    ref = torch.zeros_like(logp) if drop_reference else ref_logp
    margin_w = beta * (logp[p, w] - ref[p, w])
    if no_pair:
        return -F.logsigmoid(margin_w).mean()      # the break: no contrast
    margin_l = beta * (logp[p, l] - ref[p, l])
    return -F.logsigmoid(margin_w - margin_l).mean()


def train_dpo(comparisons, ref_logp, beta=BETA, steps=600, lr=0.05,
              log_every=0, drop_reference=False, no_pair=False, swap=False):
    """Fit the policy directly on the votes. Returns its log-probs, (4, 8).
    The policy is a table of logits here; in the paper it is a language
    model and logp is the sum of token log-probs of the whole answer."""
    torch.manual_seed(0)
    ref = torch.tensor(ref_logp, dtype=torch.float32)
    logits = ref.clone().requires_grad_(True)       # start AT the reference
    opt = torch.optim.Adam([logits], lr=lr)
    p, w, l = (torch.tensor([c[i] for c in comparisons]) for i in range(3))
    if swap:
        w, l = l, w                                 # the break: winner/loser
    for step in range(1, steps + 1):
        loss = dpo_loss(F.log_softmax(logits, dim=-1), ref, p, w, l, beta,
                        drop_reference=drop_reference, no_pair=no_pair)
        opt.zero_grad()
        loss.backward()
        opt.step()
    with torch.no_grad():
        return F.log_softmax(logits, dim=-1).numpy()

Now break it

This is the part that actually makes it stick. You debug for a living — reverse-engineering from broken behaviour is your native mode. Demo 7 runs most of these and prints them side by side, at 4000 steps so drift has time to show.

Break thisWatch for
Pass drop_reference=True — the log πref terms leave the lossIt still trains, and stops being DPO. The margin becomes plain ranking of log-probs and the fixed point is no longer the KL-constrained optimum: true 2.898 → 2.865, KL 1.183 → 1.308, recovered-reward correlation 0.763 → 0.698. On a real LM the reference is the only thing keeping text fluent — this is the road to gibberish that wins comparisons.
Set beta=0.0 in train_dpoMax |π − πref| = 1.87e-08. The policy never moves at all: β multiplies the margin, so at zero the gradient dies. This is not the RLHF page's unleashed run — the no-leash behaviour is the small-β end of the sweep, beta=0.01, KL 1.430.
Sample the votes from long_winded_policy instead of the referenceTrue reward 2.898 → 1.927 at the same KL. The derivation's on-policy assumption, violated, with no error message.
Pass no_pair=True — train on winners only, no loser termTrue reward 1.658 against the reference's 1.117 and DPO's 2.898. With nothing to contrast against it degenerates into imitating whatever the sampler produced. That's SFT, badly.
Pass swap=True — winner and loser exchangedRecovered-reward correlation +0.763 → −0.773, true reward 2.898 → −0.151. It learns the preference backwards, confidently. Check your column order.
In demo_1, change solve_objective_numerically(..., steps=4000) to 100The solver stops short of the closed form: max difference 0.000018 → 0.015708. Useful as a control: it confirms the agreement is real convergence, not a coincidence of two identical formulas.
Then read the paper — the derivation in the method section ("Deriving the DPO objective"), and Appendix A.1, which is the closed form on this page in four lines of algebra. Read the gradient discussion too: it says in one paragraph why the loss up-weights examples where the implicit reward ranks the pair wrongly. Skip the GPT-4-as-judge win-rate plots on a first pass. You'll find the paper is just the comments you've already read.

On the maths anxiety

The formula sheet: a log, an exponential, a sigmoid, a weighted average. Log and exp are inverses, which is the entire trick — the closed form has an exp in it, you take logs, and a multiplication becomes a subtraction you can cancel. Nothing else happens.

Worth sitting with: why Z(x) may disappear. Not because it's small — it's enormous and uncomputable — but because it's constant in the thing you're differencing. Same move as cancelling a common denominator, or the - max in a numerically safe softmax. Cancellation, not approximation.

What you don't need: the uniqueness proof for the KL-constrained optimum, the Plackett-Luce generalisation to rankings, any of the convex analysis. Demo 1 replaces the proof with a number you can rerun, which for engineering purposes is worth more.

Where to go next

This page gave you one thing: the inversion, and the cancellation that makes it usable. It skipped the language-model reality — token-level log-probs, memory, many epochs — and the argument over whether DPO is the right default.

Order I'd take them in: the paper's derivation → the Hugging Face fine-tuning post → the DPO-versus-PPO study. Theory, practice, then doubt. And read the RLHF page first if you haven't: DPO is a refactor, and a refactor only makes sense once you've seen the code it replaced.