GRPO: reinforcement learning without a critic

Shao et al., 2024 — DeepSeekMath

The second model everyone trains alongside the policy is a baseline predictor. You can get a baseline for free.

Download grpo_from_scratch.py — everything below runs from it. NumPy for the mechanism, plus a tiny CPU PyTorch loop for the training steps. Five and a half seconds.

I'm an AI Engineer. I've shipped models, done the MLOps, and when DeepSeek-R1 landed and everyone started saying "GRPO" I assumed it was a new algorithm I'd have to learn from scratch. It isn't. It's the policy gradient from the RLHF page with one line changed — the line that computes the baseline. That one line deletes a whole second neural network from your training job.

This page leans on two others. RLHF: fit a reward model to human votes, then climb it with a policy while a KL penalty keeps you near the model you started from. DPO: skip the climbing, because the optimum of that objective has a closed form you can turn into a supervised loss. InstructGPT: the three-stage pipeline all of it sits inside. I won't re-derive the reward model, the KL penalty or the policy gradient. GRPO keeps all three; it changes what you subtract.

Grading on a curve

Here is the policy gradient, the shape everyone writes first: 

for answer, reward in sampled:
    push_up(answer, by=reward)     # reward is 0..10, always positive

Spot the bug. Every reward is positive, so every sampled answer gets pushed up — the good ones and the bad ones alike. The differences are what you meant to learn, and they're a ripple on a large constant. It converges; practically, it's noise. So you subtract something. PPO's answer: hire a second examiner. Train a value network — a critic, roughly the size of the policy — to predict what this prompt should score, and subtract its prediction. It works, and it doubles your memory.

GRPO's answer is the one every teacher already uses:

group   = [sample_answer(prompt) for _ in range(G)]   # G answers, one prompt
rewards = [verify(a) for a in group]
for answer, reward in zip(group, rewards):
    A = (reward - mean(rewards)) / std(rewards)       # grade on the curve
    push_up(answer, by=A)                             # A is signed now

Grade the class against the class. No second model, no extra forward pass — you already had those rewards, and half the group comes out negative by construction. Nothing about the maths got better; the bill did. PPO's critic is a second network of roughly policy size with its own weights, gradients and Adam moments, and optimiser state is where RLHF memory actually goes. DeepSeekMath introduces GRPO in one setting: mathematical reasoning, where the reward is a verifier checking the final answer. If you can grade an answer with == you need no reward model, and now no critic either — the recipe DeepSeek-R1 was trained with.

TermMeansProgrammer's analogy
Baseline b(x)A number subtracted from the reward before you use it as a gradient weightThe zero point on a diff. Only the delta is signal
Advantage Areward − baseline. "Better or worse than expected for this prompt"Latency vs your p50, not raw milliseconds
Critic / value netA learned model predicting the baseline. PPO trains one alongside the policyA second service you now have to deploy, monitor and debug
Group, GG answers sampled for the same prompt. Their mean is the baselineFan out the same request G ways, compare the responses
Reference πrefThe frozen model you started fromThe main branch. Same as on the RLHF page

The one formula

Ai = ( ri − mean(r1..G) ) / std(r1..G)
loss = −(1/G) Σi Ai · log π(yi|x) + β · KL( π ‖ πref )

That's it. The real paper writes the ratio-and-clip machinery PPO uses for multiple epochs on the same batch; strip that and this is what's left. Read it as pipeline stages:

StepShapeWhat it does
y1..G ∼ π(·|x)G answersThe group. One prompt, G samples. This replaces the critic
riG scalarsReward per answer. In DeepSeekMath, a verifier's grade
− mean(r)scalarThe baseline. Costs zero parameters. Half the group goes negative
/ std(r)scalarStep size stops depending on how spread out this group happened to be
Ai · log π(yi)scalarPolicy gradient. Push up if A > 0, down if A < 0
+ β · KLscalar ≥ 0The leash from the RLHF page — but added to the loss, not to the reward

Why a baseline exists at all

The script builds a toy world in the shape of the RLHF one: 6 prompts, 12 candidate answers each, and a reward. The reward here is a verifier — a direct 0-to-10 score, no learned reward model — because that's the setting GRPO was invented for. The policy is a softmax over the 12 answers, one row of logits per prompt, so the policy gradient is exactly A * (onehot(a) − probs) and I can measure the thing everyone talks about and nobody prints: the variance of the gradient estimate. 400 independent estimates, only the advantage changes.

Don't take my word for it. Here's the actual output:

every reward in this world is POSITIVE: min 0.027   max 9.351
gradient variance, G=8, 400 independent estimates:
  no baseline           E||g - Eg||^2 =    9.7489
  group-mean baseline   E||g - Eg||^2 =    2.1146
  reduction: 4.6x
both estimators point the same way on average:
  cosine(Eg_none, Eg_group) = 0.9961
The baseline changes nothing in expectation. That's the point. Both estimators aim at the same gradient — cosine 0.9961 between their means. Subtracting a number that doesn't depend on which answer you sampled is free, because the gradient of a distribution sums to zero against a constant. All a baseline buys is variance. Every baseline argument in RL, PPO's critic included, is an argument about which free constant to pick.

The group is the baseline

One group, printed so you can do the arithmetic yourself. Prompt 2, G = 8 samples from the reference policy, drawn with replacement — so answer 3 comes up four times. A group is just G more decodes of a prompt already in the batch. 

prompt 2, G=8 sampled answers: [ 3  7  7  3  0 11  3  3]
rewards            : [6.67  2.645 2.645 6.67  4.185 2.11  6.67  6.67 ]
group mean         : 4.7828
group std          : 1.9649
advantage = (r - mean) / std :
   answer  3  r  6.670  ->  A +0.9603
   answer  7  r  2.645  ->  A -1.0881
   answer  7  r  2.645  ->  A -1.0881
   answer  3  r  6.670  ->  A +0.9603
   answer  0  r  4.185  ->  A -0.3044
   answer 11  r  2.110  ->  A -1.3604
   answer  3  r  6.670  ->  A +0.9603
   answer  3  r  6.670  ->  A +0.9603
sum of advantages  : -0.000000   (zero, by construction)

(6.670 − 4.7828) / 1.9649 = 0.9603. Do it on a phone calculator. That's the algorithm. The / std isn't cosmetic: without it your update size depends on how spread out this group's rewards happened to be, which is a property of the prompt, not the answer.

The critic you delete

Same problem, same 400-estimate measurement, three baselines: none, a fitted value baseline (the PPO-style critic, handed the exact expected reward per prompt — I'm being generous to it), and the group mean. Then 300 steps of training with each. 

gradient variance, G=8, 400 estimates:
  no baseline                       9.7489
  fitted value baseline (PPO)       2.3416
  group-mean baseline (GRPO)        2.1146

parameters:
  policy (6 prompts x 12 logits)     72
  critic MLP (8 -> 16 -> 1)         161
  group baseline                      0   <- the point

300 steps, G=8, beta=0.02:
  reference policy         true reward 3.0131
  with critic              true reward 5.5041   KL 2.1683
  group baseline           true reward 5.5390   KL 2.5183

A 161-parameter critic against a 72-parameter policy, in a toy where the critic's job is embarrassingly easy. In a real run it's a second 7B network with its own Adam state. And it buys nothing here: the group baseline gets marginally lower variance and marginally more reward, for zero parameters.

Don't over-read the toy. My critic sees a clean 8-dimensional embedding; a real one reads text and has a hard regression to solve. The honest version: the critic isn't useless, it's expensive and fallible, and the group gives you most of it for free. Two things that can be wrong in your training run become one.

How big does the group need to be?

G is the one hyperparameter GRPO adds: a straight compute-versus-variance trade, G decodes per prompt per step. The two ends of the sweep are the interesting ones. 

  G   grad variance   final true reward   note
   1          0.0000              3.0131   advantage is identically 0
   2          2.5451              5.5031
   4          1.3656              5.5278
   8          0.6290              5.5390
  16          0.2994              5.5428
  32          0.1455              5.5442

  reference policy, untrained:  true reward 3.0131

G = 1 is worth staring at. With one sample the group mean is that sample, so every advantage is exactly zero, the gradient is zero, and the policy never leaves the reference: 3.0131 in, 3.0131 out. No error, no warning, no NaN — a training run that does nothing. You need two answers before there's a curve to grade on.

At the other end variance keeps halving as G doubles, but the reward column stopped moving at G = 8. You pay four times the decodes between 8 and 32 for the fourth decimal place. Real runs sit around 8 to 64 for the same reason.

The leash, moved into the loss

GRPO keeps the KL penalty — same leash, same job as on the RLHF page: stop the policy wandering somewhere the reward signal was never validated. Sweeping β does what you'd expect. 

  beta    final true reward   final KL(pi || pi_ref)
  0.00               5.5393                 2.5190
  0.02               5.5390                 2.5183
  0.10               5.5380                 2.5114
  0.50               5.4048                 1.9776
  2.00               3.8040                 0.1162

Note what this table does not show: β = 0 is fine here, and gets the best reward. In this world the reward is a verifier and the verifier is the truth — nothing to hack. The equivalent table on the RLHF page, where the reward is a learned model, has β = 0 climbing the reward model to 1.941 while true reward falls to 1.758. Verifiable reward versus modelled reward: that's why "does GRPO need the KL term" has two right answers.

The placement is the interview detail. RLHF folds the KL penalty into the per-sample reward; GRPO adds it as a term in the loss, over the whole distribution. Sounds like bookkeeping. It isn't: 

  beta 0.1  KL in loss   true 5.5380  KL 2.5114
  beta 0.1  KL in reward true 5.4884  KL 1.8684
  beta 0.5  KL in loss   true 5.4048  KL 1.9776
  beta 0.5  KL in reward true 5.3146  KL 1.2440

Same β, visibly different leash. Folded into the reward, the penalty goes through the group baseline with everything else — mean subtracted, std divided — so the coefficient you set is not the one you get; it's been rescaled by that group's reward spread. As a loss term its gradient is taken directly against the reference, and β means what it says.

The baseline must be per-prompt

Why the paper says group and not batch. Prompt 0 is easy (average reward 6.2), prompt 5 is hard (average 1.0). Take the mean over the whole batch instead of within each prompt's group, and look at what the advantage is now measuring. 

mean advantage per prompt, one batch (easy -> hard):
 prompt  mean reward   group baseline   batch baseline
   0          6.227            0.000            3.018
   1          3.470            0.000            0.261
   2          4.783           -0.000            1.574
   3          2.138           -0.000           -1.070
   4          1.645           -0.000           -1.564
   5          0.989            0.000           -2.220

gradient variance   group 2.1146   batch 3.6445

Read the last column. On prompt 5 every sampled answer has a negative advantage, the group's best included: the number now says "how hard was this prompt", not "how good was this answer". Probability drains off everything the policy sampled and onto whatever it didn't. On prompt 0, the reverse. 

final true reward per prompt after 300 steps:
 prompt   best possible   group baseline   batch baseline
   0              9.351            9.341            9.217
   ...
   5              1.884            1.879            1.822
  overall 5.5390 (group)  vs  5.4709 (batch)

Worse on every prompt, not just the hard ones, with 70% more gradient variance. In fairness it's still unbiased in the limit — a constant is a constant — but with a finite batch each update is about the prompt's difficulty rather than the answer's quality, which is what the numbers above are. "Group relative" is the name of the algorithm, and this is why.

The code

NumPy for the mechanism — advantages, the policy gradient by hand, the variance measurements. A tiny CPU PyTorch loop for the training steps and the critic, because Adam and .backward() are not the idea. Every function under 35 lines.

Download the full script — about 220 lines of mechanism plus seven runnable demos. Then: 

python3 grpo_from_scratch.py
def policy_gradient(logits, samples, advantages):
    """samples: (P, G) sampled answer indices. advantages: (P, G) scalars."""
    probs = softmax(logits)
    grad = np.zeros_like(logits)
    G = samples.shape[1]
    for p in range(logits.shape[0]):
        for i in range(G):
            onehot = np.zeros(logits.shape[1])
            onehot[samples[p, i]] = 1.0
            grad[p] += advantages[p, i] * (onehot - probs[p]) / G
    return grad


def sample_group(logits, G, rng):
    """Sample G answers per prompt, WITH replacement -- a group. (P, G)."""
    probs = softmax(logits)
    return np.stack([rng.choice(N_ANSWERS, size=G, p=probs[p])
                     for p in range(logits.shape[0])])


# ---------------------------------------------------------------------------
# STAGE 2 -- THE WHOLE PAPER. Two lines.
#
# A critic predicts "what should this prompt score?" and you subtract it.
# GRPO says: you already sampled G answers for this prompt. Their own mean IS
# that prediction, and it costs no parameters. Divide by the group's standard
# deviation so the update size does not depend on how spread out the rewards
# happened to be. Grading on a curve within the class.
# ---------------------------------------------------------------------------
def group_advantages(r, normalise=True):
    """r: (P, G) rewards for a group. Returns (P, G) advantages."""
    adv = r - r.mean(axis=1, keepdims=True)               # the group baseline
    if normalise:
        adv = adv / (r.std(axis=1, keepdims=True) + 1e-8)  # scale invariance
    return adv

And the training step, with the KL where GRPO puts it:

        chosen = lp.gather(1, torch.tensor(samples))              # (P, G)
        pg = -(adv_t * chosen).mean()
        kl = (lp.exp() * (lp - torch.log_softmax(ref, dim=-1))).sum(-1).mean()
        loss = pg + beta * kl                                     # KL in LOSS

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living. Demos 6 and 7 run five of these for you, side by side with the working version.

Break thisWatch for
Set G=1 in train(...)Every advantage is exactly 0. — the group mean is the sample. True reward after 300 steps: 3.0131, identical to the untrained reference. Silent no-op.
Pass normalise=False to group_advantagesUpdate size tracks reward spread, not answer quality. Gradient row norms: prompt 0 (std 2.291) 1.0768, prompt 5 (std 0.601) 0.2764. Normalised they're 0.4700 and 0.4603.
Swap in batch_advantages — one mean over all promptsMean advantage becomes +3.018 on the easy prompt and −2.220 on the hard one. Variance 2.1146 → 3.6445, and every prompt ends lower.
Set beta=0.0KL climbs from 2.5183 to 2.5190 and reward is fine — because a verifier can't be hacked. Then read the same knob on the RLHF page, where β = 0 sends true reward from 2.360 down to 1.758.
Pass critic_value=np.zeros(N_PROMPTS) to grad_variance — a critic that never learned anythingA constant baseline is no baseline: variance 9.7489, exactly the no-baseline number, versus 2.3416 fitted and 2.1146 for the group mean.
In group_advantages, use r.max() instead of r.mean()Every advantage ≤ 0, so the update only ever pushes answers down. It converges to whatever the policy never sampled.
Then read the paper — but only the reinforcement-learning section, where GRPO is defined and compared against PPO, and the unified paradigm discussion at the end that puts rejection sampling, DPO and PPO in one frame. Skip the data-pipeline and pre-training half of DeepSeekMath entirely; it's a good paper about a maths corpus that happens to contain the algorithm you came for. You'll find it's just the comments you've already read.

On the maths anxiety

Less here than the notation suggests: a mean, a standard deviation, and one fact — subtracting a constant from the reward doesn't change where the gradient points on average. That fact is the whole justification for baselines, and the script demonstrates it rather than proving it (cosine 0.9961).

You don't need the policy gradient theorem. For a softmax policy the gradient of log π(a) with respect to the logits is onehot(a) − probs; for anything real it's loss.backward(). What is worth internalising is the word "variance", which RL people say constantly and which means one concrete thing: run the same estimator twice on the same policy and see how far apart the two answers land. That's grad_variance.

What I skipped: PPO's clipped ratio, which lets you take several optimiser steps on one batch of samples without the policy running away from them. Real GRPO keeps it. I take one step per batch, so the ratio is identically 1 and clipping is a no-op. Asked what GRPO inherits from PPO, that's the answer — the clipping, minus the critic.

Where to go next

This page gave you one thing: the baseline, and why deleting the critic is cheap. That's deliberately narrow. Three other places cover what I skipped.

Order I'd take them in: RLHF first for the reward model and the leash, this page second for the baseline, then Lilian Weng if the maths is nagging you and R1 if the scale is. DPO fits anywhere after RLHF — it's the other way out of the same expense.