DeepSeek-R1: reasoning nobody taught it

DeepSeek-AI, 2025

Nobody wrote the reasoning examples. Nobody rewarded length. The working-out showed up anyway, and that is the whole story.

Download deepseek_r1_from_scratch.py — everything below runs from it. NumPy for the environment, plus a tiny CPU PyTorch policy for the GRPO training loop. About 20 seconds.

Blunt about the limits first, the same way the chain-of-thought page and the RAG page are about their stand-ins. There is no language model in my script. A four-input MLP cannot demonstrate emergent reasoning in a 671-billion-parameter model, and nothing it prints is evidence about what R1 does on a maths olympiad problem.

What it can show, honestly, is the mechanism the result stands on: when a correctness-only reward is paired with a task where more steps genuinely help, a policy gradient discovers longer solutions without anybody telling it to. That claim fits on a laptop. Every number below is from that analogy, and I say so each time.

I had this paper filed under "another RLHF variant" for months. It isn't. The interesting model in it is R1-Zero, and what makes it interesting is a deletion.

You paid for correct answers and got the working-out for free

You have shipped this function:

def solve(question):
    return model(question)      # one forward pass. no locals, no loop.

And you know the fix from chain-of-thought: give it somewhere to put intermediate values, because every token it emits becomes part of its own next input. Emitted tokens are serial compute. So you write exemplars that show the working, and the continuation shows its working too.

R1-Zero is what happens when you refuse to write those exemplars:

def solve(question):
    scratch = []
    while keep_going(scratch):          # nobody wrote this loop
        scratch.append(step(question, scratch))
    return final(scratch)

reward = 1.0 if answer_is_right else 0.0     # this is all you paid for

Take a base model. Skip supervised fine-tuning entirely. Score its output with a rule — is the answer right, is it in the required tags — and run reinforcement learning. Long chains of reasoning appear on their own, including the model stopping to re-check its own working. The paper's framing is that the reasoning was incentivised, not taught.

It's the contractor you pay per correct answer with no time limit. They start showing their working, not because you asked for working, but because checking themselves is the cheapest route to the thing you're paying for. The optimiser found the same trade.

TermMeansProgrammer's analogy
R1-ZeroRL straight onto the base model, no SFT at allShipping with only an integration test and no unit tests written by hand
Rule-based rewardA grader you can read: answer correct, tags presentassert, not a learned classifier
GRPOScore a group of answers to the same prompt, compare them to each otherRanking within a batch instead of keeping a second model to predict scores
Cold startA small SFT pass before RL, added in R1 properA style guide applied before the optimiser is let loose
DistillationTrain small models on the big model's good outputsRecording a working session and replaying it, instead of rediscovering it

What it deleted from the InstructGPT pipeline

InstructGPT is three stages: supervised fine-tuning on human demonstrations, then a reward model trained on human comparisons, then PPO against that reward model. Two of those three stages need people, and the second one has the failure mode the RLHF page is entirely about — the optimiser hunting for the soft spots in a learned model of the tester.

R1-Zero deletes both.

StageInstructGPTR1-Zero
DemonstrationsHumans write ideal answers, SFT on themSkipped
RewardReward model learned from human comparisonsA rule. Is the answer right, are the tags there
OptimisationPPO with a KL penalty to the SFT modelGRPO with a KL penalty to the reference

You can only delete them because maths and code answers are checkable. There is a right answer and a script can tell. That is the boundary of the method, and it is worth saying out loud in an interview: rule-based rewards work where a unit test works.

The one formula

Two sentences of GRPO, because the GRPO page derives it properly and I'm not re-deriving it here. Sample a group of G answers to the same prompt, score them all, and use each answer's score minus the group's mean, over the group's standard deviation, as its advantage — so the group is the baseline and there is no value network to train. Then it is the ordinary policy gradient, with a KL leash back to a frozen reference.

Âi = (ri − mean(r1..G)) / std(r1..G)   ·   J = E[ Âi · log πθ(oi) ] − β · KL(πθ ‖ πref)

Read it as pipeline stages:

StepShapeWhat it does
o1..GG answers, one promptThe group. G=8 in my script. Sampled, so they differ
riscalarThe rule. Correct answer + right tags. No learned model anywhere
Âiscalar"Better or worse than my siblings?" A z-score inside the group
log πθ(oi)sum over stepsMake the above-average answers more likely, the below-average ones less
β · KLnatsThe leash. Distance from the frozen reference policy

Notice what the reward does not contain. There is no term for length, no term for "reasoning quality", no term for style. If length is going to appear, it has to appear because of the correctness term.

First, prove that thinking longer actually helps

If it doesn't, nothing else on this page means anything — so my toy task is built so that it genuinely does. Each problem is one true digit and six noisy readings of it, each reading correct 55% of the time. The policy chooses, step by step, whether to read once more or stop and answer with the majority of what it has read. There are only six real readings; step seven onwards re-reads one it already has, which is not new information and double-counts whatever noise it lands on.

Don't take my word for it. Here's the actual output:

   k   accuracy (hard, p=0.55)   accuracy (control, p=1.00)
   1            0.546                    1.000
   2            0.539                    1.000
   3            0.666                    1.000
   4            0.764                    1.000
   5            0.829                    1.000
   6            0.871                    1.000
   8            0.802                    1.000
  12            0.752                    1.000
  16            0.734                    1.000

On the hard task, more steps buy real accuracy up to the sixth, then stop and start costing. On the control task one read is already perfect, so extra steps buy exactly nothing. Same code, one parameter apart. That pair is what makes the next section honest instead of a coincidence.

The centrepiece: length rises although nothing rewards length

Now train it. Reward is 0.2 × format + 1.0 × correct. GRPO, β = 0.1, 150 iterations, eight prompts by a group of eight. Each row is 300 held-out problems so the curve is signal rather than one noisy batch. 

  HARD TASK (p=0.55, extra steps genuinely help)
   iter   mean length   accuracy
      0          0.47      0.137
     15          8.52      0.717
     30          8.12      0.767
     45          5.52      0.777
     60          6.11      0.800
     75          3.76      0.800
     90          4.46      0.783
    105          6.60      0.833
    120          4.89      0.833
    135          3.41      0.780
    150          6.15      0.860

Length goes from half a step to six, accuracy from 0.137 to 0.860, and the reward function never once mentioned length. Now the control — identical in every respect except that a single read is already correct: 

  CONTROL TASK (p=1.00, one read is already enough)
   iter   mean length   accuracy
      0          0.68      0.297
     15          2.06      0.987
     30          2.63      1.000
     45          2.18      1.000
     60          2.34      1.000
     75          2.31      1.000
     90          2.15      1.000
    105          2.12      1.000
    120          2.11      1.000
    135          2.19      1.000
    150          2.05      1.000

  hard task    length 0.47 -> 6.15   accuracy 0.137 -> 0.860
  control task length 0.68 -> 2.05   accuracy 0.297 -> 1.000
Same reward, same optimiser, same network, same seed. The control climbs to the one read it needs to answer at all and then stops, because past that point length changes nothing about correctness and there is no gradient on it. Length emerges exactly where length buys correctness, and nowhere else. That is the mechanism, and it is also the honest boundary of the claim: my script shows why the optimiser would lengthen output, not that a transformer's chain-of-thought is doing anything as tidy as this.

Reward design, which is where you will actually go wrong

Three reward functions, everything else identical. This is the section I would put on a whiteboard if someone asked me what I learned from R1. 

  reward function                  accuracy   mean length   format
  correct + format                   0.850        5.12       1.00
  format ONLY                        0.360        1.12       1.00
  correct + format + 0.05*length     0.743       16.00       1.00

Format only gets you perfectly structured output with the accuracy of a ten-sided coin. It never learned to look at the answer, because nothing ever paid it to. Every field validates. Nothing is true.

Length in the reward gets you padding. It runs to the context window and accuracy falls — 0.850 down to 0.743 — because the extra steps are re-reads, and re-reads amplify noise. You bought tokens and called it thought. This is why R1's reward has no length term: length is a symptom you want, not a target you set.

The tags, the parser, and the bug you will ship

R1's template is thinking between <think> tags and the final answer between <answer> tags, so a rule can find the answer without guessing. Mine copies that: 

<think>
step 1: read -> 9
step 2: read -> 8
step 3: read -> 9
step 4: read -> 5
step 5: read -> 9
step 6: read -> 9
</think>
<answer>9</answer>

   truth                       : 9
   parse_answer (tag-anchored) : 9
   parse_answer_broken (first  : 1   <- that's the step index
     number anywhere)

Now break the parser the way everyone breaks it once — match the first number anywhere in the output instead of the one inside the answer tags. Same model, same rollouts, two graders: 

   scored with the correct parser : 0.850
   scored with the broken parser  : 0.090

   trained AGAINST the broken parser -> real accuracy 0.330, length 1.17

The second number is the parser's opinion of the working-out, not the answer. Grade with it and you have not measured a worse model, you have measured a different string. Train against it and the reward signal is noise, so the policy learns to stop early and stops learning anything at all. A regex is a component of your training objective. Treat it like one.

The leash, swept

Same reward, same task, four values of β. KL is per trajectory, in nats, measured against the frozen reference the run started from. 

  beta     accuracy   mean length   KL from reference (nats/traj)
  0.00      0.743       16.00          16.498
  0.02      0.745       15.96          16.495
  0.10      0.850        5.12           5.113
  0.50      0.588        2.81           1.659

With no leash it runs to the context window and stays there. It still scores well — the answer is in the right tags and is often right — and the output is this: 

<think>
step 1: read -> 8
step 2: read -> 6
step 3: read -> 0
step 4: read -> 0
step 5: read -> 8
step 6: read -> 4
step 7: read -> 8
step 8: read -> 6
step 9: read -> 6
step 10: read -> 6
step 11: read -> 6
step 12: read -> 8
step 13: read -> 4
step 14: read -> 6
step 15: read -> 8
step 16: read -> 8
</think>
<answer>6</answer>

Sixteen steps of which six carry information, and a wrong answer at the end of them. β = 0.5 is the opposite failure: it never leaves the reference and never learns to think at all. β is tuned, not derived, and the number you put on a dashboard is the KL, not β — see the RLHF page, which is entirely about what an optimiser does when you take the leash off.

Distillation, and why R1 needed a cold start

The paper's other finding is that distilling R1's outputs into smaller dense models worked better than running large-scale RL on those small models directly. My version of that, matched on rollouts of the environment rather than on gradient steps, because rollouts are the expensive part: 

  teacher  (hidden=16, 150 GRPO iters)   acc 0.850   length   5.12
  kept 1051/1200 teacher rollouts -> 6446 labelled decisions

  small network, hidden=4:
    distilled from teacher, 0 new rollouts   acc 0.850   length   5.02
    GRPO directly, 18 iters (1152 rollouts)  acc 0.485   length   2.02
    GRPO directly, 150 iters (9600 rollouts) acc 0.823   length   5.47

Every successful teacher trajectory hands the student a label for every decision inside it. A reward hands it one scalar for the whole trajectory. That is the entire difference. Give the RL student eight times the rollouts and it catches up — which is exactly the trade the paper is pointing at. Matched on gradient steps instead, my two students are within noise of each other, so treat this as an analogy for their result rather than evidence of it.

And this is why R1-Zero is not the model they shipped. R1-Zero's answers were right and hard to read — poor readability, and languages mixed together inside a single chain of thought. So R1 proper adds a small cold-start supervised stage before the RL, purely to fix how the output reads. The emergence is real; it just doesn't arrive housetrained.

The code

NumPy for the environment, the majority vote and the parsers. A tiny PyTorch policy — four inputs, sixteen hidden units, three actions — CPU only, for the GRPO loop, because hand-writing backprop through 150 iterations would bury the one thing worth looking at. Every function under 25 lines.

Download the full script — about 500 lines plus 6 runnable demos. Then: 

python3 deepseek_r1_from_scratch.py
N_DIGITS = 10        # answers are digits 0-9
N_READINGS = 6       # how much genuine evidence a problem contains
K_MAX = 16           # the context window: a hard stop, not a penalty
P_CORRECT = 0.55     # per-reading accuracy on the hard task
BETA = 0.1           # default KL leash (swept in demo 5)

CONTINUE, STOP_TAGGED, STOP_RAW = 0, 1, 2


# STAGE 0 -- the environment. A problem is a truth plus six noisy readings.
# Reading k for k < 6 is fresh evidence. Reading 7, 8, 9... is a re-read of
# something already seen: no new information, and it double-counts whatever
# noise it happens to land on.
def make_problem(rng, p_correct=P_CORRECT):
    truth = int(rng.integers(N_DIGITS))
    readings = []
    for _ in range(N_READINGS):
        if rng.random() < p_correct:
            readings.append(truth)
        else:
            step = 1 + int(rng.integers(N_DIGITS - 1))
            readings.append((truth + step) % N_DIGITS)
    return truth, readings


def next_reading(rng, readings, seen):
    """Fresh evidence while there is any left, otherwise a re-read."""
    if len(seen) < N_READINGS:
        return readings[len(seen)]
    return seen[int(rng.integers(len(seen)))]


def majority(seen):
    """The answer readout. NOT learned -- the policy only decides when to stop."""
    if not seen:
        return -1
    return int(np.bincount(seen, minlength=N_DIGITS).argmax())


# STAGE 1 -- what the policy sees. Four numbers: how much context is spent, how
# decisive the current vote is, how big the leading count is, and a bias term.
# Deliberately tiny, so nothing interesting can hide inside the network.
def features(seen):
    k = len(seen)
    if k == 0:
        return [0.0, 0.0, 0.0, 1.0]
    counts = np.sort(np.bincount(seen, minlength=N_DIGITS))[::-1]
    return [k / K_MAX, (counts[0] - counts[1]) / k, counts[0] / k, 1.0]


# STAGE 2 -- the output format. R1 asks for thinking between <think> tags and
# the final answer between <answer> tags, so that a rule -- not a learned
# reward model -- can grade it. STOP_RAW emits the same content with no tags,
# which is the degree of freedom the format reward exists to remove.
def render(seen, answer, tagged):
    steps = "\n".join(f"step {i + 1}: read -> {v}" for i, v in enumerate(seen))
    if tagged:
        return f"<think>\n{steps}\n</think>\n<answer>{answer}</answer>"
    return f"{steps}\nthe answer is {answer}"


ANSWER_RE = re.compile(r"<answer>\s*(\d+)\s*</answer>")
FORMAT_RE = re.compile(r"<think>.*</think>\s*<answer>.*</answer>", re.S)


def parse_answer(text):
    """Pull the final answer out of the tags. Returns None if it isn't there."""
    m = ANSWER_RE.search(text)
    return int(m.group(1)) if m else None


def parse_answer_broken(text):
    """The bug everyone ships once: first number ANYWHERE. Grabs the working."""
    m = re.search(r"(\d+)", text)
    return int(m.group(1)) if m else None


# STAGE 3 -- the reward functions. R1-Zero's is rule-based: accuracy plus
# format, no learned reward model, because a learned one invites the optimiser
# to go hunting for its soft spots (see the RLHF page). Note what is NOT here:
# any term that mentions length.
def reward_correct_format(text, truth, n_steps, parser=parse_answer):
    return 0.2 * format_ok(text) + 1.0 * (parser(text) == truth)


def reward_format_only(text, truth, n_steps, parser=parse_answer):
    return 0.2 * format_ok(text)                      # never mentions the answer


def reward_with_length(text, truth, n_steps, parser=parse_answer):
    return reward_correct_format(text, truth, n_steps, parser) + 0.05 * n_steps


# STAGE 5 -- GRPO, in one function. Two sentences: sample a group of G answers
# for each prompt, score them, and use (r - mean of the group) / (std of the
# group) as the advantage, so no value network is needed -- the group is the
# baseline. Then the usual policy gradient, plus a KL leash back to the
# reference policy. Full derivation on the GRPO page; this is the same
# estimator, in 20 lines.
def grpo_step(policy, ref, opt, roll, rewards, group, beta):
    r = torch.tensor(rewards, dtype=torch.float32).view(-1, group)
    adv = (r - r.mean(1, keepdim=True)) / (r.std(1, keepdim=True) + 1e-6)
    adv = adv.view(-1)

    x = torch.tensor(roll["feats"], dtype=torch.float32)
    logp = torch.log_softmax(policy(x), dim=-1)
    chosen = logp[torch.arange(len(roll["acts"])), torch.tensor(roll["acts"])]

    # sum the log-probs of each trajectory, weight by that trajectory's advantage
    per_traj = torch.zeros(roll["n"]).index_add_(
        0, torch.tensor(roll["owner"]), chosen)
    pg = -(per_traj * adv).mean()

    # the leash: KL(policy || reference) at every state actually visited
    with torch.no_grad():
        ref_logp = torch.log_softmax(ref(x), dim=-1)
    kl = (logp.exp() * (logp - ref_logp)).sum(-1)
    kl_traj = torch.zeros(roll["n"]).index_add_(
        0, torch.tensor(roll["owner"]), kl)

    opt.zero_grad()
    (pg + beta * kl_traj.mean()).backward()
    opt.step()
    return float(kl_traj.mean())

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living — reverse-engineering from broken behaviour is your native mode. Every failure below is one my script prints today.

Break thisWatch for
Train with reward_format_only — pay for the tags, never for the answer Format score 1.00, accuracy 0.360 against 0.850. Perfectly structured output that is wrong two times in three. Every field validates, nothing is true
Train with reward_with_length — add 0.05 * n_steps to the reward Mean length jumps to 16.00, the context window, and accuracy drops to 0.743. Nineteen-line traces of which six lines carry information
Pass parser=parse_answer_broken — match the first number anywhere instead of inside <answer> The same rollouts score 0.090 instead of 0.850, because it is grading the step index in the working. Train against it and real accuracy lands at 0.330
Set BETA = 0.0 — take the KL leash off Length pins at 16.00, KL from the reference reaches 16.498 nats, accuracy falls to 0.743. Output that still scores well and reads as padding
Run the whole thing with p_correct=1.0, so extra steps cannot help Length stays at 2.05 from iteration 15 to 150 while accuracy sits at 1.000. Nothing emerges, because there is nothing for length to buy
Set N_READINGS = 16 so every step is fresh evidence The plateau in demo 1 disappears and the accuracy column keeps rising to k=16 — and the length-in-reward run stops looking like padding, because now it isn't
Then read the paper — the R1-Zero sections first, especially the reward-design discussion and the table tracking response length during training, which is the real version of my demo 2. Then the cold-start and distillation sections. Skip the benchmark tables on a first pass; they are the least transferable part. You'll find the paper is mostly the comments you have already read.

On the maths anxiety

There is less here than the notation suggests. The advantage is a z-score: subtract the mean of eight numbers, divide by their standard deviation. You have written that in a monitoring dashboard. It exists so the policy is told "better or worse than your siblings" rather than "your score was 0.83", which is what removes the need for a separate value network.

The policy gradient is one derivative you can take on trust: nudge up the log-probability of things that went better than average, nudge it down for things that went worse. The KL term is a distance between two probability distributions, measured in nats, and in code it is literally (p * (log p − log q)).sum(). That's it. Nothing in this paper needs a proof to be used correctly.

What does need care is not maths at all — it's the reward function and the parser that feeds it. Those are ordinary software, they are wrong in ordinary ways, and they are where the whole method succeeds or quietly fails. The question you will be asked in an interview is not how GRPO's advantage is derived. It's "how did you know the model got better, and not just better at your grader?"

Where to go next

This page gave you one angle — the emergence mechanism and the reward-design traps. These fill in the parts I skipped: the real training runs, the algorithm's derivation, and the wider family of reasoning models.

Order I'd take them in: Raschka first for the map, then the R1-Zero sections of the paper, then DeepSeekMath if the advantage term still feels like a magic trick, then Open-R1 when you want to run one yourself.