Chinchilla: how big a model should you train?

Hoffmann et al., 2022

A budget allocation problem, not a research result. This is the one paper on this site you can put in a funding request.

Download scaling_laws_from_scratch.py — everything below runs from it. NumPy only.

I'm an AI Engineer. I've sized clusters, argued about instance types, and signed off on training runs — and for years I could not have told you, on paper, whether the money was better spent on a bigger model or on more data. I assumed someone upstairs had done the maths. Then I read this paper and found the maths is a two-term curve fit and an argmin over one variable. Twenty lines. The industry got it wrong for two years anyway.

It's sizing infrastructure against a fixed monthly bill

You've done this. Fixed budget, two things to buy:

budget = 10_000        # per month, and it is not going up
# spend it on bigger instances, or on more of them?
for n_instances in range(1, 100):
    size = budget / n_instances
    print(n_instances, size, throughput(n_instances, size))

Chinchilla is that loop. The budget is FLOPs. The two things you buy are parameters (N) and training tokens (D). And there is one hard constraint tying them together, so picking one picks the other: 

C = 6 * N * D                 # your whole training bill, in FLOPs
for N in candidate_sizes:
    D = C / (6 * N)           # tokens you can still afford at that size
    print(N, D, loss(N, D))   # pick the row with the smallest loss

That is the entire paper. Everything else is measuring loss(N, D) carefully enough that the argmin means something.

TermMeansProgrammer's analogy
NParameter countInstance size. Bigger costs more per hour, forever.
DTraining tokens seenHow long you leave the job running.
CTraining FLOPsThe invoice. Fixed before you start.
Compute-optimalThe (N, D) with lowest loss at that CThe row your sweep picks.

Why this overturned Kaplan

Two years earlier, Kaplan et al. ran the same experiment and got a different answer: spend almost everything on model size. Their fitted rule was N growing like C0.73 against D like C0.27 — ten times the compute means roughly five times the model and only two times the data. GPT-3 (175B parameters, 300B tokens) is that advice, executed.

Chinchilla redid it with over 400 runs and got N and D both growing like C0.5. Equal split. Grow them together.

Kaplan wasn't sloppy, it was set up differently, and two differences do most of the work. Kaplan used a single cosine learning-rate schedule length across runs of different lengths, which makes short runs look worse than they are; Chinchilla tuned the schedule to each run's actual token count. Kaplan also counted non-embedding parameters. If you want the argument in full, the Chinchilla paper's discussion section spells it out — I'm summarising, not adjudicating.

The one formula

L(N, D) = E + A / Nα + B / Dβ   subject to  C = 6ND

Read the loss as three bills added together:

TermShapeWhat it does
EscalarThe floor. What no model in the fit gets below — call it the irreducible part.
A / NαscalarToo small. The penalty for not having enough parameters to represent the function.
B / DβscalarToo little data. The penalty for not having seen enough text.
6NDscalarThe constraint. Fix C, and choosing N fixes D.

The two penalty terms pull opposite ways under the constraint, so the curve is a bowl and the bowl has a bottom. That bottom is the paper.

E, A, B, α and β are the paper's fitted values, not mine. I did not derive a single one of them. The script hardcodes two published sets and re-solves the optimisation they imply — that part is arithmetic you can check.

Where the 6 comes from

Per parameter, per token: 2 FLOPs forward (a multiply and an add), 4 backward (gradient with respect to the activations, gradient with respect to the weights). 2 + 4 = 6. That's the whole derivation. It ignores attention's own quadratic-in-sequence-length term, which stays small while the context is short relative to model width — true for every model in this paper, less true for long-context models today.

Don't take my word for it. Here's the actual output:

model          params   tokens  tok/param       FLOPs
GPT-3          175.0B   300.0B        1.7    3.15e+23
Gopher         280.0B   300.0B        1.1    5.04e+23
Chinchilla      70.0B     1.4T       20.0    5.88e+23  ✓
LLaMA-7B         7.0B     1.0T      142.9    4.20e+22

Gopher and Chinchilla cost about the same to train. One is four times the size of the other. That comparison is the paper's headline experiment, and you can compute both sides of it with a pocket calculator.

Sweep the split, find the bottom

Fix C at Chinchilla's own training budget, walk N from 1B to 2T, and let the constraint hand you D. Then just read the loss column. 

budget C = 5.76e+23 FLOPs (roughly what Chinchilla spent)

  N (params)  D (tokens)  tok/param  predicted loss
        1.0B       96.0T    96000.0          2.1904
        5.0B       19.2T     3840.0          2.0501
       20.0B        4.8T      240.0          1.9911
       50.0B        1.9T       38.4          1.9758
       72.0B        1.3T       18.5          1.9744  ← bottom
      100.0B      960.0B        9.6          1.9755
      200.0B      480.0B        2.4          1.9850
      500.0B      192.0B        0.4          2.0138
        2.0T       48.0B        0.0          2.1008  ← all size, no data

argmin over 20000 candidate splits (REFIT coefficients):
  N* = 72.3B params
  D* = 1.3T tokens
  D*/N* = 18.4 tokens per parameter
  loss  = 1.9744 nats/token
  Chinchilla as actually built: 70.0B params, 1.4T tokens, 20.0 tok/param

72.3B against the 70B they built. That is the paper predicting its own model from a curve fit, and it is the moment the whole thing stopped feeling like numerology to me.

An honest wrinkle: which coefficients

Here is the part I nearly got wrong. Chinchilla used three methods — a training-curve envelope, an IsoFLOP sweep, and the parametric fit above — and reported that all three agreed on roughly twenty tokens per parameter. But if you type the parametric fit's printed coefficients into the optimisation, you don't get twenty. You get about ninety.

That's not me making an arithmetic error; it's a known discrepancy, and a 2024 replication refitted the same data and reported coefficients that do reproduce the headline. My script carries both sets and prints them side by side, because "the published numbers don't quite reproduce the published conclusion" is exactly the kind of thing a page like this should say out loud rather than quietly paper over. 

same sweep, ratio only, across five budgets:
  budget C  REFIT tok/param  PAPER tok/param
     1e+21             21.6             50.1
     1e+22             20.4             62.6
     1e+23             19.2             78.1
     6e+23             18.4             92.6
     1e+25             17.1            122.0
Everything below uses the refitted coefficients. They reproduce the paper's own headline and its own model; the printed ones don't. The takeaway that survives either way: N and D should grow together, and the ratio is a small two-digit number, not one.

Kaplan vs Chinchilla on the same money

Kaplan gives an exponent, not a constant, so to draw their curve I anchor it on a real model built under that advice — GPT-3, 175B on 300B tokens. The exponent is theirs; the anchor is mine, and it is the weakest assumption on this page. 

  budget C   Kaplan N  Kaplan D    t/p     loss   Chin. N   Chin. D    t/p     loss
     1e+21       2.6B     63.5B   24.2   2.3056      2.8B     60.0B   21.6   2.3055
     1e+22      14.1B    118.2B    8.4   2.1452      9.0B    184.2B   20.4   2.1411
     1e+23      75.7B    220.1B    2.9   2.0444     29.5B    565.8B   19.2   2.0321
     6e+23     271.9B    353.1B    1.3   1.9925     72.3B      1.3T   18.4   1.9744
     1e+24     406.7B    409.8B    1.0   1.9792     95.9B      1.7T   18.1   1.9597
     1e+25       2.2T    763.1B    0.3   1.9357    312.1B      5.3T   17.1   1.9117

at C = 5.76e+23:
  Kaplan wants a model 3.8x bigger, fed 3.8x fewer tokens
  and pays 0.0180 nats/token for it

Look at the t/p column under Kaplan. It falls below one. At a big enough budget, Kaplan's rule recommends a model with more parameters than training tokens — parameters you cannot afford to train. That is the failure mode, stated as a number.

Same spend, worse model. The interview question here is "why was GPT-3 undertrained?", and the answer is not "they were careless" — it is "the scaling law they were following put the money in the wrong place, and it took 400 runs to show it."

Chinchilla against Gopher

The paper's proof is one head-to-head. Same lab, same data pipeline, roughly the same training bill, radically different split. 

Gopher       N= 280.0B  D= 300.0B  tok/param=   1.1  C=5.040e+23  loss=1.9996
Chinchilla   N=  70.0B  D=   1.4T  tok/param=  20.0  C=5.880e+23  loss=1.9739

compute ratio Chinchilla/Gopher = 1.167  (i.e. the same training bill)
predicted loss gap = +0.0257 nats/token in Chinchilla's favour

Chinchilla is a quarter the size and beat Gopher across the paper's benchmark suite. The loss numbers above are what this fit predicts, not measured benchmark scores — don't quote them as the paper's results. The compute ratio is the load-bearing number: 1.167, near enough the same money.

Why LLaMA ignored all of this

Chinchilla optimises the training bill. You also pay to serve the thing, roughly 2 FLOPs per parameter per generated token, no backward pass, every day, forever. Put that in the constraint and the answer moves: 

C_total = 6*N*D_train + 2*N*D_inference
total budget (train + serve) = 5.76e+23 FLOPs

 inference tokens       N*   D* train  tok/param  train loss
             none    72.3B       1.3T       18.4      1.9744
           100.0B    69.0B       1.4T       19.7      1.9751
             1.0T    50.1B       1.6T       31.6      1.9806
            10.0T    15.8B       2.7T      174.1      2.0128
            50.0T     4.4B       5.0T     1140.2      2.0772
           100.0T     2.4B       6.8T     2822.3      2.1231

Read it top to bottom. The more tokens you expect to serve, the smaller the model you should train and the longer you should train it. The training loss column gets worse — that is the point. You are deliberately accepting a worse-trained model because you will pay for it a billion times at inference.

Chinchilla answers "best model I can train". LLaMA answered "best model I can run". LLaMA-7B saw about a trillion tokens — roughly 143 per parameter, seven times past compute-optimal. Not a contradiction of this paper; a different objective. That argument is worked through on the LLaMA page.

Where the fit stops being trustworthy

The paper's runs covered roughly 70M to 16B parameters and 5B to 400B tokens. The formula does not know that. Ask it about a model a million times larger and it answers with four decimal places and total composure. 

         N         D  tok/param  predicted loss
     70.0M      1.4B       20.0          3.6587
     70.0B      1.4T       20.0          1.9739
     10.0T    200.0T       20.0          1.8440
   1.0e+16   2.0e+17       20.0          1.8195
   1.0e+19   2.0e+20       20.0          1.8174
      1.0K     20.0K       20.0        101.1373

the fit's floor is E = 1.8172: at infinite N and D the formula
says loss -> 1.8172 nats/token and never lower.
loss(1e19 params, 2e20 tokens) - E = 1.97e-04

Two things in that block are nonsense and the formula flags neither. A thousand-parameter model does not have a loss of 101. And E is a fitted constant, not a measured entropy of language — it is wherever the curve happened to flatten inside one box of runs, and treating it as a law of nature is how scaling-law arguments go wrong on slide four. Outside the box you fitted, this is arithmetic, not evidence.

The code

Pure NumPy. No torch, no training, nothing to wait for — the paper is a curve fit and an argmin, and both are a few lines. Every function under 15 lines.

Download the full script — about 75 lines of core code plus seven runnable demos. It finishes in well under a second. Then: 

python3 scaling_laws_from_scratch.py
import numpy as np

def compute_flops(N, D):
    """Training FLOPs for N parameters over D tokens."""
    # 2 FLOPs forward + 4 backward, per parameter per token. That is the
    # whole derivation of the 6. It ignores attention's own quadratic term,
    # which is small while context is short relative to model width.
    return 6.0 * N * D


def tokens_for_budget(N, C):
    """Invert C = 6ND: how many tokens does budget C buy at size N?"""
    return C / (6.0 * N)


# Two coefficient sets, BOTH published, NEITHER derived here.
#   PAPER -- Hoffmann et al. 2022, the "Approach 3" parametric fit as printed.
#   REFIT -- Besiroglu et al. 2024 replication, which refitted the same data
#            because the printed coefficients do not reproduce the paper's
#            own headline. Demo 2 shows the gap. REFIT is the default.
#
#   L(N, D) = E + A/N^alpha + B/D^beta
#     E            an irreducible floor: no model in the fit beats it
#     A/N^alpha    the penalty for being too small
#     B/D^beta     the penalty for not having seen enough text
PAPER = dict(E=1.69,   A=406.4,  B=410.7,   alpha=0.34,   beta=0.28)
REFIT = dict(E=1.8172, A=482.01, B=2085.43, alpha=0.3478, beta=0.3658)


def loss(N, D, coef=REFIT):
    """Predicted loss in nats per token for N params trained on D tokens."""
    return (coef["E"] + coef["A"] / N ** coef["alpha"]
            + coef["B"] / D ** coef["beta"])


def optimal_split(C, coef=REFIT, n_lo=1e6, n_hi=1e15, steps=20000):
    """Return (N*, D*, loss*) minimising loss under budget C = 6ND."""
    # THE WHOLE PAPER: minimise L(N, D) subject to 6ND = C.
    # One budget, one free variable: pick N, and D falls out of the
    # constraint. So sweep N and take the argmin. No calculus needed.
    Ns = np.logspace(np.log10(n_lo), np.log10(n_hi), steps)
    Ds = tokens_for_budget(Ns, C)
    Ls = loss(Ns, Ds, coef)
    i = int(np.argmin(Ls))
    return Ns[i], Ds[i], Ls[i]


# The correction the paper deliberately did NOT make. Serving costs about
# 2 FLOPs per parameter per generated token, no backward pass. If you expect
# to serve D_inf tokens, the budget you really control is
#     C_total = 6*N*D_train + 2*N*D_inf
# Same minimisation, different constraint. This is the LLaMA argument.
def optimal_split_with_inference(C_total, D_inf, n_lo=1e6, n_hi=1e15,
                                 steps=20000):
    """Minimise loss subject to 6*N*D_train + 2*N*D_inf = C_total."""
    Ns = np.logspace(np.log10(n_lo), np.log10(n_hi), steps)
    Ds = (C_total - 2.0 * Ns * D_inf) / (6.0 * Ns)   # may go negative
    ok = Ds > 0
    Ls = np.where(ok, loss(Ns, np.where(ok, Ds, 1.0)), np.inf)
    i = int(np.argmin(Ls))
    return Ns[i], Ds[i], Ls[i]


# Kaplan et al. (2020) fitted N_opt ~ C^0.73: spend it on size. Their paper
# gives an exponent, not a constant, so anchor on a model built under that
# advice -- GPT-3. The exponent is theirs; the anchor is mine.
KAPLAN_EXPONENT = 0.73
ANCHOR_N, ANCHOR_D = 175e9, 300e9
ANCHOR_C = compute_flops(ANCHOR_N, ANCHOR_D)


def kaplan_split(C):
    """Kaplan-style (N, D) for budget C, anchored on GPT-3."""
    N = ANCHOR_N * (C / ANCHOR_C) ** KAPLAN_EXPONENT
    return N, tokens_for_budget(N, C)

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living — reverse-engineering from broken behaviour is your native mode, and this script is fast enough that you can break it fifty times in a coffee break.

Break thisWatch for
In demo_2, replace optimal_split(C) with the Kaplan split at the same CN jumps 72.3B → 271.9B, D collapses 1.3T → 353.1B, loss goes 1.9744 → 1.9925. Same invoice, worse model.
In demo_5, set every D_inf to 0.0Every row returns 72.3B / 18.4 tok/param. You've silently optimised the training bill for a model you'll serve 100T tokens through.
Add (1e19, 2e20) to demo_6's list — it's already there, delete the READ THIS and believe the number1.8174, four decimals, two orders of magnitude past any run in the fit. 1e3 params reports a loss of 101.1373.
In demo_7(b), pass total parameters to compute_flops instead of active onesC reads 2.820e+23 instead of 7.800e+22 — a 3.6x overstatement. See the MoE page for why only the active experts do work.
In demo_7(c), pin D_fixed and grow N alone10B → 10T params moves the loss 2.1096 → 1.9638 and it never gets below 1.9493. The B/D term does not care how big your model is.
Swap REFIT for PAPER as the default in loss()The optimum ratio goes from 18.4 to 92.6 tokens per parameter, and the fit no longer predicts the model the paper built.
Then read the paper — the three estimation approaches and the table that predicts optimal N and D for a given budget. Skip the full benchmark appendix unless you need a specific score; it's a leaderboard, not an argument. You'll find the paper is mostly the comments you've already read, plus the enormous amount of GPU time it took to earn them.

On the maths anxiety

This paper needs less maths than any other on this site. A power law (A / N**alpha — a thing that shrinks as N grows, quickly at first, then slowly). An argmin (find the smallest row). Multiplication. That's it. There is no gradient anywhere in my script.

The one genuinely mathematical move is curve fitting: they ran 400+ models, wrote down a functional form with five unknowns, and used an optimiser to pick the five numbers that best explain the observations. If you've ever fitted a latency curve to decide an autoscaling threshold, that's the same act at the same level of rigour.

What this paper actually demands is judgement, not algebra: knowing that a fit is only valid inside the range it was fitted on, and that compute-optimal training is not the same objective as cheap serving. Both of those are engineering instincts you already have from capacity planning.

Where to go next

This page gave you the optimisation and the budget arithmetic, and skipped almost everything about how the three estimation approaches were actually run. These fill that in.

Order I'd take them in: Chinchilla → de Vries → the replication → Kaplan. Chinchilla gives you the rule, de Vries tells you when to break it, the replication tells you how much precision the rule really has, and Kaplan is history you only need once you care why anyone believed otherwise.