Hallucination and grounding: the model was never trying to be right

Ji et al., 2022 — with Kalai & Vempala, 2023

The word makes it sound like a malfunction. It isn't one, and that's why the fix you're waiting for is not coming.

Download hallucination_from_scratch.py — everything below runs from it. NumPy only. There is no language model anywhere in it. What it runs is the mechanism: a store that keeps facts as overlapping features instead of as records, so answering is reconstruction rather than lookup. Every demo says so on the line above, the way the chain-of-thought and RAG pages label their stand-ins.

I shipped a support assistant that invented a refund policy. Beautifully. Correct tone, correct format, a policy that has never existed. I filed it as a bug, went looking for the setting that turns it off, and slowly worked out there isn't one — because nothing in that model's training ever rewarded being right. It was rewarded for producing likely text. On that question the likely text and the true text were different strings, and it picked the one it was trained to pick.

Everybody had told me "it hallucinates", which frames it as a component misbehaving. It isn't. It's the component working.

It's an autocomplete that read everything and remembers nothing

You know the difference between these two:

facts["pellucid"]["licence"]     # a record. Exact, or KeyError.
autocomplete("pellucid is licensed under ")   # a continuation. Always something.

The first has a row per fact, so a fact seen once is as retrievable as a fact seen a million times. The second has no rows. It has weights — one smear of overlapping features that every fact was added into during training — and asking it a question doesn't look anything up. It reconstructs a plausible answer out of the smear. That is the whole page. Reconstruction is confident by construction, because a blend always produces a value. There is no cache miss. There is no KeyError. The None branch you'd have written by hand does not exist in the architecture.

TermMeansProgrammer's analogy
HallucinationFluent output the source or the world doesn't supportA cache that synthesises a value on a miss and never tells you
GroundingPutting the fact in the context so it's copied, not recalledPassing the config in instead of hard-coding a default
CalibrationStated confidence matching observed accuracyA p99 estimate that's actually right 99% of the time
AbstentionRefusing to answer below a confidence thresholdReturning 503 instead of a guess

The one formula, and what's missing from it

θ* = argmaxθ Σt log Pθ(xt | x<t)

Read it as pipeline stages:

StepWhat it isWhat it does
x<tthe prefixEverything so far — prompt plus what's been generated
Pθ(xt | …)a distributionHow likely each next token is, given that prefix
loga logSo multiplying probabilities becomes adding them. Float hygiene
Σta sum over tokensScore the whole corpus, one token at a time
argmaxθthe training loopMove the weights until the corpus looks as likely as possible
There is no term in that formula for "true". Not hidden in the log, not implied by the sum. The objective is a fit to what the corpus said. Where the corpus was right, fitting it produces right answers as a side effect — and that side effect is the entire basis of the thing you shipped.

Ji et al. split it into intrinsic hallucination, output that contradicts its own source, and extrinsic, output the source neither supports nor contradicts. Worth having straight: the second kind is the one nobody's eval catches, because it isn't wrong against the passage, it's unverifiable from it.

Reconstruction is not recall, and confidence can't tell

The script builds a store of 24 facts about eight fictional projects. Not a dict — one 64×64 matrix that every fact is added into, weighted by how often the toy corpus said it. Popular projects appear hundreds of times, obscure ones once. One project, tessera, is deliberately a fork of lantern: its feature vector is mostly lantern's, because a store like this has no notion of "different project", only "nearby vector".

Don't take my word for it. Here's the actual output:

  dict lookup   ('lantern','licence')  -> MIT   ('tessera','licence') -> GPL-3
  the store     ('lantern','licence')  -> MIT   conf 1.000   truth MIT    correct
  the store     ('tessera','licence')  -> MIT   conf 0.999   truth GPL-3  WRONG

  licence question, projects ordered by how loud the corpus was:
  lantern   count  220  answered MIT         conf 1.000  truth MIT         correct
  quarry    count   90  answered Apache-2.0  conf 0.970  truth Apache-2.0  correct
  halyard   count   40  answered MIT         conf 0.991  truth MIT         correct
  marlin    count   12  answered Apache-2.0  conf 0.886  truth Apache-2.0  correct
  cobble    count    3  answered BSD-3       conf 0.624  truth BSD-3       correct
  pellucid  count    1  answered MIT         conf 0.398  truth GPL-3       WRONG
  tessera   count    1  answered MIT         conf 0.999  truth GPL-3       WRONG

  over all 24 questions: accuracy 0.708
  mean confidence when RIGHT 0.950 | when WRONG 0.701
  highest confidence on a WRONG answer: 1.000
  lowest confidence on a RIGHT answer : 0.624

Read the last two lines together. The ranges overlap. Confidence is not useless — 0.950 when right against 0.701 when wrong is real signal — but there's a wrong answer at 1.000 sitting above a right answer at 0.624, so any threshold you pick cuts through both piles. Confidence measures how much support the reconstruction found. Support correlates with truth and is not truth.

The likely continuation and the true one are different strings

This is the central claim, so here it is as one printed table. Left: what the objective prefers. Right: what's true.

  question                  most likely   p       true         p
  --------------------------------------------------------------
  tessera.licence           MIT         0.999   GPL-3       0.000   <- differ
  tessera.language          Python      1.000   Rust        0.000   <- differ
  pellucid.port             6379        0.488   9200        0.340   <- differ
  cobble.language           Python      0.641   Java        0.337   <- differ
  lantern.licence           MIT         1.000   MIT         1.000   ✓

  and the corpus prior the objective is actually fitting:
    P(answer | licence ) argmax = MIT         0.804  all [0.804 0.186 0.005 0.004]
    P(answer | language) argmax = Python      0.568  all [0.568 0.351 0.074 0.006]
    P(answer | port    ) argmax = 8080        0.728  all [0.728 0.174 0.073 0.025]

80.4% of licence mentions in the toy corpus are MIT, and the objective's job is to reproduce that. Answering "MIT" for a project it saw once isn't a failure of the job — it's the job, on a question where doing the job gives the wrong answer. A fluent wrong answer is a correctly executed argmax over the wrong quantity.

Calibration makes some hallucination unavoidable

This is the part almost nobody covers, and it's the one that changed how I argue in planning meetings. A calibrated model is one whose stated confidence matches its observed accuracy: of the things it says at 70%, about 70% are true. That's a property you want. Now take facts that appeared exactly once in training. A calibrated model can't assign them probability 1, because the evidence isn't there — so it spreads probability over plausible alternatives, and some of that lands on statements that are false. Kalai & Vempala (2023) turn that into a theorem: for facts with no more support than a single mention, a calibrated model's hallucination rate is bounded below by roughly the fraction of such singleton facts. Not "tends to". Must.

The script calibrates its toy the standard way, one temperature scalar, then sweeps the abstention threshold:

  raw:  mean confidence 0.877  vs accuracy 0.708  -> overconfident by 0.169
  calibrated temperature 1.35 -> mean confidence 0.712  vs accuracy 0.708 ✓

  facts attested exactly once in the corpus: 6/24 = 0.250

  threshold   answered   refused   errors   err|answered   wrong/100q
  --------------------------------------------------------------------
    0.00       24/24      0       7       0.292        29.17
    0.40       20/24      4       4       0.200        16.67
    0.50       18/24      6       3       0.167        12.50
    0.60       16/24      8       3       0.188        12.50
    0.70       15/24      9       3       0.200        12.50
    0.80       12/24     12       3       0.250        12.50
    0.90        8/24     16       2       0.250         8.33
    0.95        0/24     24       0           -         0.00
    0.99        0/24     24       0           -         0.00

Look at the last two rows before you celebrate the zero. Zero errors, because zero answers. That's the product decision this section is about, and it's why the column worth watching is wrong/100q rather than err|answered — the second flatters a system that has quietly stopped working.

Abstention trade-off at five confidence thresholds Five bars, one per confidence threshold, each 24 questions wide. As the threshold rises from 0.00 to 0.95 the right-answer segment shrinks from 17 to 0, the wrong-answer segment from 7 to 0, and the refused segment grows from 0 to 24. 24 questions, split by what the system did green outline = right · red outline = WRONG · dashed = refused 0.00 17 · 7 · 0 0.50 15 · 3 · 6 0.80 9 · 3 · 12 0.90 6 · 2 · 16 0.95 0 · 0 · 24 threshold right · WRONG · refused Zero errors arrives on the same row as zero answers.
The knob has one setting that eliminates error, and it also eliminates the product. Every number here is printed by demo 3.

To be explicit: my toy's numbers are not evidence about any real model, and this sweep shows only the shape of the trade. The theorem is theirs; the eight fictional projects are mine.

Grounding: stop asking it to recall

The engineering answer isn't clever. If the failure is that recall reconstructs, don't ask it to recall — put the fact in the context, where copying it is the likely continuation. That's all retrieval-augmented generation does. The RAG page builds the retriever half; this is what the retriever is for.

Two paths to an answer, and where a wrong answer enters each Left path: the question goes straight to the weights and out as an answer, with one error entry point, A, where the fact was rare or absent in training. Right path: the question goes to a retriever, then passages, then the weights, with three entry points, B, C and D: the retriever misses the answer, the prior overrides the passage, and two passages disagree. Path 1 — from the weights Path 2 — retrieved, then answered question the weights Afact rare or absent→ reconstructed answer 1 place to go wrong,no way to check it question retriever Bno passage holdsthe answer passages Dtwo passagesdisagree the weights Cprior overridesthe passage answer 4 places to go wrong, 3 of them checkable
Grounding doesn't take the weights out of the path — it adds a stage in front of them. That's why the number of failure modes goes up while the error rate goes down.

In the script, grounding is one line: a fact present in the context adds a fixed boost to that answer's score before the softmax. Same store, same questions, with and without:

  from the weights alone : 17/24 = 0.708
  with the fact in context: 21/24 = 0.875
  errors removed by grounding: 4

  residual errors WITH the fact present: 3
    tessera.licence: said MIT (conf 0.996), passage said GPL-3
    tessera.language: said Python (conf 0.978), passage said Rust
    tessera.port: said 8080 (conf 0.997), passage said 6379

    GAMMA 0.5 -> grounded accuracy 0.875
    GAMMA 1.0 -> grounded accuracy 0.875
    GAMMA 3.0 -> grounded accuracy 0.875
    GAMMA 6.0 -> grounded accuracy 0.917
    GAMMA 10.0 -> grounded accuracy 0.958
    GAMMA 20.0 -> grounded accuracy 1.000

The residual block is the half everyone omits. Three answers contradict a passage that was right there in the context, at confidence 0.98 and above, and all three are the fork — the case where the prior is loudest. In a real system that prior is "what the internet mostly says", and it does not stand down because you pasted a document above it. The GAMMA sweep is the honest version of "just use RAG". How hard a model prefers context over its own weights is a continuous property set by pretraining and instruction tuning, and you mostly don't control it. It is not a boolean you flipped on when you added a vector database.

Three ways a grounded answer is still wrong

Same store, a simulated retriever, all 24 questions, three failure modes measured separately:

  failure mode                  wrong / asked   rate
  ----------------------------------------------------
  passage lacks the answer        10 /  24     0.417
  answer contradicts passage       3 /  24     0.125
  two passages disagree            6 /  24     0.250

  Mode (a) is a RETRIEVAL bug. The generator behaved correctly:
  it was handed nothing about the subject and filled the gap.
    tessera.licence with only a lantern passage -> MIT (conf 1.000), truth GPL-3
Mode (a) is a retrieval bug that gets logged as a model bug. It's the most common of the three here at 0.417, and it's exactly what the RAG page's break-it table already shows you doing to yourself with k=1 and oversized chunks. Fix the retriever, not the prompt. Only the middle mode is arguably the model's fault; the third is a corpus problem wearing a model's face.

Detecting it without a ground truth

In production you don't have the answer key. What you do have is the ability to sample the same question several times. If the model reconstructed from thin evidence the samples scatter; if it's copying something solid they agree. That's self-consistency, and it costs you k times the tokens. 

  mean agreement when the modal answer is RIGHT 0.941
  mean agreement when it is WRONG               0.714
  correlation(agreement, correctness) = 0.522  -> a real signal

  and the cases it misses -- agreement >= 0.875 and still wrong: 3
    tessera.licence: said MIT 8/8 times, truth GPL-3
    tessera.language: said Python 8/8 times, truth Rust
    tessera.port: said 8080 8/8 times, truth 6379

A real signal and a real ceiling in one block. Agreement separates the two populations — and then the fork answers wrongly 8 times out of 8, because its error is structural. The features say lantern, so every sample says lantern. Sampling variance cannot detect a bias. Use it as a flag, never as a gate. Finding out how well it works on your data is an eval problem, and that's the evals page's job — including the part where a metric that looks fine on aggregate is useless on the slice you care about. On the published side, Min et al.'s FActScore is the sane framing: split a long answer into atomic claims and score each against a source, rather than grading the paragraph as a unit.

A citation is not a proof. A model can produce a citation that exists, is retrievable, and does not support the sentence attached to it. Citations make an answer checkable by a human, which is worth a lot, and they do not make it true. Product copy that says "grounded, with citations" claims the first thing and is heard as the second.

The code

Pure NumPy. No torch, no autograd, no training, and no language model — just a store, a reconstruction and a softmax, so you can watch the numbers move. Every function under 15 lines.

Download the full script — 412 lines plus 6 runnable demos. Then: 

python3 hallucination_from_scratch.py
# One row per project: (licence, count), (language, count), (port, count).
# `count` is how many times that fact appeared in the toy corpus.
CORPUS = {
    "lantern":  (("MIT", 220),        ("Python", 190), ("8080", 140)),
    "beacon":   (("MIT", 180),        ("Go", 160),     ("8080", 120)),
    "quarry":   (("Apache-2.0", 90),  ("Python", 85),  ("5432", 60)),
    "halyard":  (("MIT", 40),         ("Rust", 35),    ("6379", 25)),
    "marlin":   (("Apache-2.0", 12),  ("Go", 10),      ("9200", 8)),
    "cobble":   (("BSD-3", 3),        ("Java", 2),     ("5432", 2)),
    "pellucid": (("GPL-3", 1),        ("Java", 1),     ("9200", 1)),
    "tessera":  (("GPL-3", 1),        ("Rust", 1),     ("6379", 1)),
}


# tessera is a fork of lantern. A store like this has no notion of "different
# project"; it only has "nearby vector", so lantern's answers leak into it.
# This is the one hand-placed similarity in the script, and it is the whole
# confidently-wrong case.
SUBJ_VEC["tessera"] = 0.88 * SUBJ_VEC["lantern"] + 0.12 * SUBJ_VEC["tessera"]


def key(subject, relation):
    """Question -> a feature vector. The relation contributes less than the
    subject, so two questions about the same project overlap most."""
    k = SUBJ_VEC[subject] + 0.35 * REL_VEC[relation]
    return k / np.linalg.norm(k)


def build_store(facts):
    """Superpose every fact into one matrix. No per-fact slot exists after."""
    M = np.zeros((D, D))
    for s, r, o, c in facts:
        M += np.log1p(c) * np.outer(key(s, r), OBJ_VEC[o])   # log: frequency
    return M                                                  # with diminishing
                                                              # returns


def answer(subject, relation, passages=(), gamma=GAMMA, temp=TEMP):
    """Returns (best answer, confidence, full distribution over candidates)."""
    recalled = STORE.T @ key(subject, relation)      # a blend, not a record
    cands = RELATIONS[relation]
    scores = np.array([OBJ_VEC[o] @ recalled for o in cands])

    for p_subj, p_rel, p_obj in passages:            # grounding: copy, not recall
        if p_rel == relation and p_obj in cands:
            match = 1.0 if p_subj == subject else 0.6   # off-subject counts less
            scores[cands.index(p_obj)] += gamma * match

    probs = softmax(scores / temp)
    i = int(np.argmax(probs))
    return cands[i], float(probs[i]), probs

Now break it

This is the part that makes it stick. Reading won't do it. You debug for a living — and every row here is a mistake I have watched a team make in a review.

Break thisWatch for
Ask for a fact in neither the weights nor the context: set marlin's language count to 0 in CORPUS, then call answer("marlin", "language", passages=[])Python, conf 0.649, truth Go. The fact is gone from the store and it still answers — with the corpus's favourite language, at a confidence that looks like every other answer. It never returns "I don't know", because that branch does not exist.
Trust demo 5's mode-(a) output: answer("tessera", "licence", [("lantern", "licence", "MIT")])MIT (conf 1.000), truth GPL-3. The passage was about a different project. This is a retrieval bug, not a model bug — the generator was handed nothing about tessera and filled the gap, which is what it is for.
Take confidence as correctness: gate on conf >= 0.95 and ship whatever passestessera.licence sails through at 0.999 and is wrong, while cobble.licence is right at 0.624 and gets thrown away. The ranges overlap; the threshold cuts through both.
Set demo 3's threshold to 0.95 and read only the error column0/24 answered, 24 refused, 0 errors. A perfect hallucination rate on a product that answers nothing — the metric-gaming failure your dashboard will not flag.
Judge groundedness by whether the answer sounds like the passage: compare demo 4's residual lines to their passagessaid MIT (conf 0.996), passage said GPL-3 — same question, same format, same vocabulary, opposite fact. Style similarity is not entailment, and an LLM judge asked "does this look supported?" scores it as supported.
In demo 5, drop mode (b)'s copy strength from gamma=0.8 to gamma=0.3"answer contradicts passage" goes 0.125 → 0.208, and at 0.1 it reaches 0.250 — the passage is present and correct every time. How hard a model prefers context over its own prior is a continuous property you inherit, not a switch you flipped by adding retrieval.
Then read the papers — Ji et al. for the taxonomy (the intrinsic/extrinsic definitions and the metrics section are the useful parts; the per-task chapters are a reference, not a read), and Kalai & Vempala for the calibration argument, which is short and mostly readable if you skip the proofs. You'll find they're the comments you've already read, with real models where my 64×64 matrix is.

On the maths anxiety

There's less here than the word "theorem" suggests. The objective is a sum of logs — a log so multiplying probabilities becomes adding them, the same trick you use to stop a float underflowing. The store is an outer product added into a matrix, the crudest possible associative memory, about four lines. Reading from it is a matrix-vector multiply, which is a weighted average wearing a hat.

Calibration needs one idea: bucket your predictions by stated confidence, then check observed accuracy in each bucket. If the 0.7 bucket is right 70% of the time, you're calibrated. That's a groupby and a mean. The Kalai & Vempala result is a real proof and you don't need to follow it to use it — you need the statement, which is that calibration and a zero hallucination rate on rare facts are incompatible.

What separates engineers on this topic isn't maths. It's knowing that "reduce hallucination" decomposes into retrieval quality, context adherence, abstention policy and eval design — four different teams' worth of work — and being able to say which one your incident belongs to. The interview question is "how would you stop the model making things up", and the answer that lands is: you can't stop it. You can move where the facts come from, measure what's left, and decide what confidence you'll refuse below.

Where to go next

This page gave you one thing: why hallucination is the objective working rather than failing, in code you can break. It skipped the taxonomy, the metrics, the detection literature, and everything about how real models behave.

Order I'd take them in: Kalai & Vempala first, because it reframes everything and takes an evening; then Weng for the map; then FActScore when you need a number; Ji as a reference you keep open. Read the RAG page before any of them if you haven't — grounding is the whole engineering answer here, and evals is how you find out whether it worked. GPT is where the next-token objective comes from, and chain-of-thought is the other page about what that objective does and doesn't buy you.