Evaluation: the part everyone skips

Zheng et al., 2023 — with Liang et al., 2022 and Sainz et al., 2023

You can't unit-test "is this answer good", so people ship on vibes and find the regression in production.

Download evals_from_scratch.py — everything below runs from it. NumPy only.

Blunt about the script first, the same way the chain-of-thought page is about its state machine and the RAG page is about its retriever. There is no language model in this one. The systems, the judges and the graders are explicit probability models, five lines each, and the code says so on the line above. That is not a shortcut — it is the only way any of this can be measured. You cannot check whether an evaluation recovers the truth unless you set the truth yourself first.

The attention page ends with a list of what actually separates AI engineers from people who can recite the transformer architecture. Eval design is on that list. This page is that debt paid. Every other page on this site proves its claim by running code. This is the one where that stops being easy, and that is exactly why it gets skipped.

It's a test suite. That's the whole metaphor.

You have written this a thousand times:

def test_parse_date():
    assert parse("2024-01-02") == date(2024, 1, 2)   # passes or it doesn't

The assertion is exact, deterministic and free. Now the model version. The output is a paragraph, "correct" is a matter of degree, and the only thing on hand that can grade it at any scale is another model: 

def test_summary():
    out = model(prompt)
    assert judge(out) == "good"      # a flaky assertion, and it costs money

Nothing about your instincts changes here. A flaky test is worse than no test, because it teaches you to ignore red. A test that passes because the answer leaked into the fixture is worse still, because it teaches you to trust green. Both of those failures have exact analogues in LLM evaluation, they both have numbers attached, and this page prints them.

TermMeansProgrammer's analogy
Golden setYour fixed set of test cases with known-good answersThe test fixtures, checked into the repo
LLM-as-a-judgeA model grading another model's outputAn assertion that is itself a flaky service call
Pairwise / arenaShow a grader two answers, ask which is betterA diff review instead of an absolute code score
ContaminationBenchmark items that were in the training dataThe test leaked into the fixture. It always passes.
SliceScore broken out per category of inputPer-module test results, not just "42 passed"

Three papers, one problem

HELM (Liang et al., 2022) is the argument against a single number. Its contribution is breadth and multi-metric reporting: scenarios crossed with metrics — accuracy, calibration, robustness, bias, toxicity, efficiency — side by side rather than collapsed into one leaderboard column. In test-suite terms: stop reporting "97% passing", report the matrix.

Judging LLM-as-a-Judge (Zheng et al., 2023) is the paper that made grading-by-model respectable. On their benchmarks — MT-Bench and Chatbot Arena — they report strong agreement between capable judge models and human preference, on the order of the agreement between two humans. That is their reported finding on their data, not a general guarantee about your task. The same paper documents the failure modes carefully: position bias, verbosity bias, self-enhancement bias, and weakness on maths and reasoning grading. Read it as "this works, here is exactly how it lies".

Contamination (Sainz et al., 2023, and a growing pile since) is the unresolved one. Benchmarks are public text on the internet. Training corpora are scraped from the internet. Detecting the overlap reliably, especially for a model whose training data you cannot see, is still an open problem. Nobody has fixed this. Assume your favourite public benchmark is partly contaminated and you will rarely be wrong.

The one formula

Not a model formula. This is the one you actually need, and almost nobody computes it before sizing a golden set:

gap ± 1.96 √( pA(1−pA)/n + pB(1−pB)/n )

Read it as pipeline stages:

StepSizeWhat it does
pA, pBscalarsMeasured pass rates. Each test case is a coin flip; this is the coin's bias
p(1−p)/nscalarVariance of one measured rate. Note the n underneath — this is the only knob you own
sum, then √scalarTwo independent systems: add variances, take the root. That's the standard error of the difference
× 1.96scalarTurns it into a 95% interval. The 1.96 is a lookup, not a derivation
gap ±intervalIf it straddles 0, your eval cannot tell the two systems apart

How big does the golden set have to be?

Set up two systems where I know the answer: A is right 72% of the time, B is right 78%. A real, worth-shipping, six-point improvement. Now run the eval at four sizes, four thousand times each, and count how often it gets the ranking backwards.

Don't take my word for it. Here's the actual output:

true accuracy:  system A = 0.72   system B = 0.78   true gap = +0.06

 n      mean measured   typical 95% CI        picks the
        gap             on the gap            WORSE system
 --------------------------------------------------------------
   20     +0.059       [-0.208, +0.328]        26.8%
   50     +0.060       [-0.109, +0.229]         20.7%
  200     +0.060       [-0.025, +0.145]          6.8%
 1000     +0.059       [+0.022, +0.098]        0.1% ✓

Look at the middle column, not the left one. The measured gap is right on average at every size — the eval isn't biased. It's just so wide at n=20 that the interval spans everything from "B is 21 points worse" to "B is 33 points better", and one run in four crowns the worse system. That's a coin flip with extra steps and a dashboard.

At n=1000 the interval finally excludes zero. That is what a six-point difference actually costs. If your golden set is twenty hand-written cases — and mine was, for a year — you have been sampling noise and telling a story about it.

Smaller effects need quadratically more cases. The √n is doing the damage: halving the effect you want to detect costs four times the test set. This is why "we'll just eyeball a few examples" fails at exactly the point where it matters — the late-stage changes are always the small ones.

Your judge is right 80% of the time. What does that cost?

Model the judge as simply as it can be modelled: it repeats the true verdict with probability a, and flips it otherwise. That has a closed form, and the closed form is the whole lesson:

measured_gap = (2a − 1) × true_gap

A judge that agrees with a human 80% of the time doesn't add noise around your number. It multiplies your number by 0.6, every time, in the same direction. Two systems 20 points apart in reality: 

true gap between A and B, graded by a perfect human: 0.2003

 judge      measured    measured    measured   gap kept
 agreement  acc(A)      acc(B)      gap
 --------------------------------------------------------------
  1.00      0.5998      0.8001      +0.2003     100.0% ✓
  0.95      0.5904      0.7712      +0.1808      90.3%
  0.90      0.5823      0.7369      +0.1547      77.3%
  0.80      0.5559      0.6791      +0.1232      61.5%
  0.70      0.5398      0.6223      +0.0825      41.2%
  0.60      0.5223      0.5626      +0.0403      20.1%
  0.50      0.5017      0.4951      -0.0066      -3.3%

Two things fall out of that table. First: a mediocre judge doesn't tell you lies at random, it compresses — every system drifts toward 50% and every gap shrinks. Your leaderboard gets flatter and flatter until everything looks the same, which people read as "we've plateaued". Second, and worse: the compression interacts with the section above. A gap squeezed to 60% of its size needs roughly three times the test cases to reach the same confidence.

This is why "measure your judge against human labels" is not bureaucracy. You need a to know what your numbers mean, and it costs a few hundred human-labelled cases, once.

The three biases, each one measurable

Give two answers of identical quality to a judge. A fair judge returns 50.0% and nothing else. Anything above that is manufactured. Here are the three biases the LLM-as-a-judge paper documents, each simulated as a single additive nudge, then mitigated: 

Two answers of IDENTICAL quality. A fair judge must return 50.0%.

 bias                       naive     after mitigation  residual
 --------------------------------------------------------------
 position (+0.10 to first)  60.1%     49.9%            0.0012 ✓
 length   (+0.12 to longer) 62.2%     49.7%            0.0031 ✓
 self-preference (+0.08)    57.9%     49.7%            0.0032 ✓

The mitigations are all boring, which is the good news:

BiasMitigationWhy it works
Position — whoever is shown first wins moreAsk twice, positions swapped, average the two verdictsA constant bonus to "first" appears on both sides and cancels exactly
Length — the longer answer winsCompare within a length bucket, or normalise for lengthIf both answers are the same length there is nothing for the bias to key on
Self-preference — the judge likes its own familyA panel of judges from different families, verdicts averagedTwo opposite leans of roughly equal size cancel in the mean

The residual column is the point. It's what the mitigation did not remove, it is never exactly zero, and it belongs in your report next to the win rate. "System A wins 58% of the time" and "system A wins 58% of the time, and our judge manufactures 8 points of that when it grades its own family" are different claims.

The swap is the cheapest win in this entire page. It doubles your judge cost and removes a ten-point systematic lie. If you do one thing from here, do that.

Contamination: the test leaked into the fixture

Hold a model's true ability fixed at 0.55 — it never gets better, not once, anywhere in this table — and vary only the fraction of the benchmark that appeared in its training data. Leaked items aren't tests any more. They're dictionary lookups. 

true held-out ability, held CONSTANT at 0.55

 leaked fraction   benchmark score   clean score   inflation
 --------------------------------------------------------------
        0%            0.5500           0.5500       +0.0000 ✓
        5%            0.5715           0.5500       +0.0215
       15%            0.6145           0.5500       +0.0645
       30%            0.6790           0.5500       +0.1290
       50%            0.7650           0.5500       +0.2150
       80%            0.8940           0.5500       +0.3440

A leaderboard-topping model on a 30% contaminated benchmark:
  model X: contaminated 0.6790   clean 0.5500
  model Y: contaminated 0.6200   clean 0.6200
  ranking on the benchmark: X wins. Ranking on clean data: Y wins.

That last block is the shape of the problem in three lines. X looks excellent and is ordinary. Y looks ordinary and is better. The benchmark ranks them backwards and no amount of re-running fixes it, because leakage is bias, not noise. It only ever points up. Noise averages out; this doesn't.

Nothing in that table measures any real model or any real benchmark — it's arithmetic on numbers I chose. What it shows is the shape: a 30% leak buys thirteen free points of nothing.

The engineering response is dull and it works: hold out a private eval set that has never been published anywhere, date-stamp it after the model's training cutoff where you can, and treat every public benchmark number — including your own — as an upper bound.

Why "rate it 1 to 5" is worse than "which is better"

Same underlying quality, two ways of asking. A is truly 0.60, B is truly 0.70. The absolute grader maps quality onto a 1–5 scale and carries a per-session calibration offset — today it's generous, next week it's stern — because a scale needs an anchor it doesn't have. You rate A in one session and B in another, so that offset lands straight in your gap. The pairwise grader sees both answers in the same session and only has to say which is better, so the offset sits on both sides and cancels. 

true quality: A = 0.60, B = 0.70. 400 repeats of a 30-case eval.
grader noise 0.7 per item, calibration drift 0.6 per session.

  absolute 1-5   mean gap +0.445   std 0.799   picks B  68.5%
  pairwise A/B   mean rate 0.659   std 0.085   picks B  94.0% ✓

  signal-to-spread: absolute 0.56, pairwise 1.88

Note what does not fix the absolute score: more test cases. The drift is shared by every item in the session, so averaging over items never touches it. The comparison isn't less noisy because the grader tried harder — it's less noisy because the question has fewer degrees of freedom.

This is the same reason the reward model in RLHF is fitted to A-vs-B votes rather than star ratings, and it's why the arena format in the LLM-as-a-judge paper is pairwise. Humans and models are both bad at absolute scales and both fine at comparisons. Design around it.

The aggregate went up. Do not ship it.

This is what an eval harness is actually for, and it's the section I'd want on a whiteboard in an interview. A deploy improves four categories and destroys the fifth: 

 category      before    after     delta
 --------------------------------------------------------------
 summarise     0.6970   0.8270   +0.1300
 extract       0.7155   0.8520   +0.1365
 classify      0.7480   0.8645   +0.1165
 translate     0.6875   0.7995   +0.1120
 code          0.7025   0.3425   -0.3600  <-- REGRESSION
 --------------------------------------------------------------
 AGGREGATE     0.7101   0.7371   +0.0270   <-- ships green

 aggregate 95% CI on the change: [+0.0146, +0.0394] -- 'significant improvement'

The aggregate rose. The confidence interval excludes zero. By every rule in the first half of this page, that is a real, statistically significant improvement, and it is also a disaster: one user in five just lost more than half their success rate. The green number and the red one are the same deploy.

The fix is four lines — group by category, compare per group, fail the build if any group drops past a threshold. It's the difference between "42 tests passed" and a per-module report. Nobody would accept the former from CI and everybody accepts it from an eval.

Slice before you aggregate. Every aggregate score is a weighted average that can hide an arbitrarily bad category, and the rarer the category, the better it hides. Your smallest slice is where your worst regression lives.

The code

Pure NumPy. No torch, no model, no network. Judges and graders are explicit probability models — which is precisely what lets the script know the truth and catch the eval getting it wrong. Every function under 15 lines.

Download the full script — about 130 lines of mechanism plus eight runnable demos. Then: 

python3 evals_from_scratch.py
import numpy as np

RNG = np.random.default_rng(0)


def run_eval(p, n, rng):
    """Return the pass/fail vector for one system on n cases."""
    # A system with true accuracy p, run on n test cases, is n Bernoulli
    # draws. That is the entire statistical model of your golden set.
    return (rng.random(n) < p).astype(int)


def diff_ci(a, b):
    """95% CI on (mean(a) - mean(b)) for two pass/fail vectors."""
    # Standard error of one proportion: sqrt(p(1-p)/n).
    # Two independent systems: add the variances, then sqrt.
    # If the interval straddles 0, your eval cannot tell them apart.
    pa, pb = a.mean(), b.mean()
    se = np.sqrt(pa * (1 - pa) / len(a) + pb * (1 - pb) / len(b))
    d = pa - pb
    return d, d - 1.96 * se, d + 1.96 * se


def judged_accuracy(truth, agree, rng):
    """Grade a pass/fail vector through a judge with agreement `agree`."""
    # The judge repeats the true verdict with probability `agree`, else
    # flips it. Closed form: measured = agree*p + (1-agree)*(1-p),
    # i.e. measured_gap = (2*agree - 1) * true_gap.
    keep = rng.random(len(truth)) < agree
    return np.where(keep, truth, 1 - truth).mean()


def pairwise_judge(qa, qb, len_a, len_b, own_a, own_b,
                   pos=0.0, length=0.0, self_pref=0.0, rng=RNG):
    """P(judge picks A) -> a 0/1 vote. A is always shown first."""
    p = 0.5 + 0.5 * (qa - qb)          # quality: a fair judge stops here
    p += pos                            # position: A is on top, A gets a bump
    p += length * np.sign(len_a - len_b)
    p += self_pref * (own_a - own_b)
    return (rng.random() < np.clip(p, 0.0, 1.0)) * 1


def swapped_win_rate(n, qa, qb, len_a, len_b, own_a, own_b,
                     pos, length, self_pref, rng):
    """A's win rate, averaged over both presentation orders."""
    # A constant bonus to whoever is first appears on both sides of the
    # average and cancels. Same trick as running an A/B test both ways.
    wins = 0.0
    for _ in range(n):
        v1 = pairwise_judge(qa, qb, len_a, len_b, own_a, own_b,
                            pos, length, self_pref, rng)          # A first
        v2 = pairwise_judge(qb, qa, len_b, len_a, own_b, own_a,
                            pos, length, self_pref, rng)          # B first
        wins += 0.5 * (v1 + (1 - v2))
    return wins / n


def contaminated_score(true_ability, leak, memorised=0.98):
    """The score a contaminated benchmark reports for a true ability."""
    # `leak` of the items were in training: on those the model scores
    # `memorised`. On the rest it scores what it can actually do.
    return leak * memorised + (1 - leak) * true_ability


def absolute_rating(q, offset, noise, rng):
    """Quality in [0,1] -> a 1-5 rating, session offset plus noise."""
    raw = 1 + 4 * q + offset + rng.normal(0, noise)
    return float(np.clip(np.round(raw), 1, 5))


def pairwise_vote(qa, qb, offset, noise, rng):
    """Same grader, same session: which of these two is better?"""
    a = 1 + 4 * qa + offset + rng.normal(0, noise)
    b = 1 + 4 * qb + offset + rng.normal(0, noise)   # offset cancels here
    return int(a > b)

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living — reverse-engineering from broken behaviour is your native mode, and every row below is a mistake I have personally shipped.

Break thisWatch for
In demo_1, keep only n = 20The eval crowns the worse system 26.8% of the time. A golden set of 20 is a coin with a dashboard.
In demo_3, call naive(pos=0.10) and never swapped_win_rate60.1% win rate for two identical answers. Ten points manufactured purely by which one is printed first.
In demo_6, print only the AGGREGATE row+0.0270 with a CI of [+0.0146, +0.0394]. Ships green while the code slice falls 0.7025 → 0.3425.
In demo_7, report scores[winner] instead of the fresh set0.7608 reported against a true 0.70. A free +0.0616, from 20 candidates that were all identical.
In demo_8, compare agree=0.92 against agree=0.84A −0.0320 "regression" in a system that did not change. You moved the ruler, not the model.
In judged_accuracy, set agree=0.50Measured gap goes to −0.0066 on a true gap of +0.2003. The leaderboard flattens and every system ties.
Then read the papers — but selectively. In the LLM-as-a-judge paper: the section on the judge's limitations and biases, and the agreement numbers against human votes. Skip the prompt appendices on a first pass. In HELM: the taxonomy of scenarios and metrics in the front half. Do not attempt the full results grid, it's over a hundred pages and it is a reference, not a read.

On the maths anxiety

There is more maths on this page than on any other one I've written, and it's also the easiest maths on the site. Nothing here goes past what a decent A-level stats course covers: a proportion, its variance p(1−p)/n, adding two variances, and multiplying by 1.96 because that's the number in the table for 95%. No calculus. No linear algebra. No gradients.

You don't need the derivation of the standard error to use it, the same way you don't need to have written a TCP stack to reason about retries. What you do need is the reflex: whenever someone shows you a difference between two numbers, ask how many samples it came from. That reflex is worth more in this job than the whole transformer architecture, and it takes an afternoon to acquire.

What this page skips: real inter-annotator agreement statistics (Cohen's kappa and friends), multiple-comparison correction when you're testing twenty slices at once, Elo and Bradley–Terry rating systems as used by the arenas, and every practical question about actually writing good test cases — which is a content problem, not a statistics problem, and is genuinely harder than anything here. The question you will be asked in an interview is not "derive the standard error". It is "how do you know your change helped?", and the answer starts with a number of test cases.

Where to go next

This page gave you the statistics of evaluation, in code you can break. It deliberately skipped the craft: how to write test cases that catch real failures, and how anyone runs this at scale.

Order I'd take them in: Husain first, because it will change what you do on Monday; then the LLM-as-a-judge limitations section; then HELM and the contamination paper when someone quotes a benchmark number at you and you want to know what it's worth.