Retrieval-Augmented Generation for Knowledge-Intensive NLP

Lewis et al., 2020

The open-book exam, explained as the search engine it actually is.

Download rag_from_scratch.py — everything below runs from it. NumPy only. There is no language model in the loop: the "generation" step assembles the prompt and prints it.

I've shipped RAG. Twice. Embedding model, vector database, LLM, done. And every time it answered wrong, I blamed the LLM. Then I read the paper and found out the LLM is the small half. The half you build, tune and debug is the retriever — and nobody had told me it's a search engine with a matrix in it.

RAG is an open-book exam

You know this:

answer = llm(question)     # closed book: whatever the weights happen to remember

The paper's twist is to hand the model the right page first:

hits   = index.search(embed(question), k=3)   # open the book to the right pages
prompt = f"Context:\n{hits}\n\nQuestion: {question}"
answer = llm(prompt)                          # answer from the page, not memory

That's the whole paper's beating heart. The model is the student, the index is the book, and the prompt is the page you slide across the desk. Or: a cache in front of a database. The weights are the database — slow to change, expensive to rebuild. The index is the cache — swap it any time. The paper literally does this: swap the 2018 Wikipedia index for a 2016 one and the answers to "who is the current…" about world leaders change to match. Nothing retrained. The book changed, the student didn't.

TermMeansExam analogy
Query xThe question, as typedThe exam question
Passage zOne chunk of the corpusOne page of the textbook
IndexEvery passage as a vector, stackedThe textbook's index
Retriever pη(z | x)Scores a passage against the questionFinding the page
Generator pθ(y | x, z)The LLM, given passage plus questionThe student, page open
Parametric memoryWhat's in the weightsWhat the student memorised
Non-parametric memoryWhat's in the indexThe textbook

Before this, a closed-book model like T5 kept every fact in its weights, and the extractive systems (DPR with a reader, REALM) fetched passages but could only copy a span out of one. RAG fetches, then generates. That set the state of the art on three open-domain QA benchmarks, ahead of the 11-billion-parameter closed-book T5 and the extractive retrievers.

The one formula

p(y | x) ≈ Σz ∈ top-k pη(z | x) · pθ(y | x, z), where pη(z | x) ∝ exp(d(z)T q(x))

Read it as pipeline stages, retriever first:

StepShapeWhat it does
d(z)(768,) × 21M rowsEvery passage through BERT once. Frozen. Stored in FAISS.
q(x)(768,)The question through a second BERT — the only part of the retriever that trains
d(z)T q(x)scalarSimilarity. A dot product, the same one as attention
top-kk passagesargsort, keep the best k (5 or 10 in the paper)
pη(z | x)k numbers, sum to 1Softmax over the k scores: how much to trust each page
pθ(y | x, z)one answer per passageBART, given [passage ; question], writes an answer
Σone answerWeighted average of the k answers, weights from the retriever

Embedding without a model

The paper's retriever is DPR: two BERT encoders (see the BERT page), one for questions, one for passages, trained so a question and the passage that answers it land close together. I don't have a trained BERT in a NumPy script and don't want one hiding the pipeline. So the script uses the oldest trick in search: count the words, weight rare words higher (tf-idf), hash each word to a slot in a fixed-width vector, divide by the length.

Honest for the shape — one fixed-size vector per chunk, compared by dot product, exactly what FAISS sees. Dishonest for the meaning: DPR knows "owns" and "owned" are one idea; a hash table doesn't. That costs us in a minute.

Here's one chunk becoming a vector:

chunk 4:  Lantern's default job timeout is 45 minutes.
tokens:   ['lantern', 'default', 'job', 'timeout', '45', 'minutes']
buckets:  [279, 223, 248, 402, 299, 675]

vector shape: (1024,)   nonzero entries: 6 of 1024
L2 norm: 1.000 ✓
  bucket  279  lantern   idf 1.38  weight 0.200
  bucket  223  default   idf 3.40  weight 0.491
  bucket  248  job       idf 2.01  weight 0.291
  bucket  402  timeout   idf 2.70  weight 0.391
  bucket  299  45        idf 3.40  weight 0.491
  bucket  675  minutes   idf 3.40  weight 0.491

vocabulary: 118 distinct tokens -> 1024 buckets, 8 hash collisions

"lantern" is in 14 of the 21 chunks so it earns the smallest weight; "default", "45" and "minutes" are in one chunk each so they carry the vector. That's all tf-idf is. The corpus is 21 sentences about a made-up job scheduler called Lantern — invented, so nothing can be fact-checked against the world; consistent, so every question has one right chunk.

Retrieval is one matrix multiply

Stack the chunk vectors into a matrix. Embed the question the same way. index @ q gives one score per chunk; argsort gives the top k. That's the retriever. At 21 rows it's NumPy; at 21 million it's FAISS with an approximate nearest-neighbour structure so you don't scan every row. Same object.

Three questions, top-3 each. * marks the answer chunk:

Q: What is Lantern's default job timeout?
   query tokens: ['lantern', 'default', 'job', 'timeout']
* #1  0.720  chunk  4  Lantern's default job timeout is 45 minutes.
  #2  0.315  chunk  5  A job that exceeds its timeout is retried at most twice.
  #3  0.251  chunk  0  Lantern is the internal job scheduler at Fennel Systems...

Q: Which team owns Lantern?
   query tokens: ['team', 'owns', 'lantern']
* #1  0.342  chunk 12  Lantern is owned by the Platform team.
  #2  0.174  chunk 13  The Platform team's on-call rotation is one week long.
  #3  0.172  chunk 14  Priya Raman is the current tech lead of the Platform team.

Q: How long are failed jobs kept in the dead-letter table?
   query tokens: ['long', 'failed', 'jobs', 'kept', 'dead', 'letter', 'table']
* #1  0.766  chunk 19  Failed jobs are kept in the dead-letter table for 30 days.
  #2  0.398  chunk  0  Lantern is the internal job scheduler at Fennel Systems...
  #3  0.175  chunk 13  The Platform team's on-call rotation is one week long.

Three for three — but look at the middle one. "owns" matched nothing, because the corpus says "owned". The hit came from "team" plus "lantern", scored a limp 0.342, and the runners-up got there on "team" alone. That gap between word overlap and meaning is the whole reason DPR exists. The pipeline is identical either way.

The prompt is the page you slide across the desk

Take the top-k chunks, number them, put the question underneath. That string is what a real system sends to the LLM. The script prints it instead of sending it, because from here the LLM's job is the easy half. 

Answer using only the context below.
If the answer is not in the context, say you do not know.

Context:
[1] Lantern's default job timeout is 45 minutes.
[2] A job that exceeds its timeout is retried at most twice.
[3] Lantern is the internal job scheduler at Fennel Systems: every batch job
    at Fennel is submitted to Lantern, queued by Lantern, given a timeout by
    Lantern, and retried by Lantern when it fails; failed jobs that keep
    failing are kept by Lantern in the dead-letter table, and failed jobs stay
    there until they expire.

Question: What is Lantern's default job timeout?
Answer:

(103 words. An LLM call goes here. There is none in this script.)

Now the same question with the retriever unplugged:

Context:
(none)

Question: What is Lantern's default job timeout?
Answer:
The model will answer this. Confidently. With a number. Lantern is private; the weights have never seen it. That confident wrong number is the "hallucination" everyone blames on the LLM. It was a retrieval miss. The interview question is "why does my RAG hallucinate?" and the answer, most days, is "it doesn't — your retriever does."

Two RAGs: the paper's and the one you'll ship

Nobody says this out loud: the system everyone calls RAG is not the system in the paper. The paper trains the retriever and the generator together. What gets shipped is a frozen pipeline.

PieceThe paper (2020)What gets shippedThis script
RetrieverDPR bi-encoder; query side fine-tunedA frozen embedding model, or BM25Hashed tf-idf, no training
IndexFAISS over 21M Wikipedia passages, 100 words eachA vector databasenp.stack, 21 rows
GeneratorBART-large, fine-tunedA frozen LLM behind an APIPrints the prompt
Passages → modelOne at a time, k answers averagedAll k pasted into one promptAll k pasted into one prompt
Training signal(question, answer) pairs; no label for which passage to fetchNoneNone

The training is the clever part. Fine-tune the query encoder and BART jointly on question–answer pairs; the document encoder and the index stay frozen, because re-embedding 21 million passages every step is not a thing. The gradient still reaches the retriever through pη(z | x): if a passage helped the generator get the answer, the query encoder learns to pull that kind of passage closer. Nobody labels "this was the right passage". It's learned from the answer alone.

Two variants. RAG-Sequence picks one passage for the whole answer, runs the generator once per passage, averages the k answers. RAG-Token re-picks the passage at every generated token, so one answer can draw on several documents. On the QA benchmarks they're within a couple of points, RAG-Sequence usually ahead; RAG-Token wins Jeopardy-question generation, where an answer needs facts from more than one page. The frozen pipeline is neither — one prompt, one call — and the script codes that, because that's what you'll debug. The generator itself is a transformer decoder: the attention page with a mask.

Three ways the retriever fails quietly

Every one of these shows up as "the LLM was wrong". None of them is the LLM. The script breaks each on purpose.

1. Skip the normalisation and length wins

Delete the divide-by-norm and a dot product stops being a cosine. A long chunk has a long vector, and a long vector wins comparisons it has no business winning. The corpus has a README paragraph — 37 tokens, mentions everything, answers nothing. Watch it take over: 

Q: What is Lantern's default job timeout?
   cosine #1:  chunk  4  (0.720)      raw dot #1:  chunk  0  (26.88)
Q: Which team owns Lantern?
   cosine #1:  chunk 12  (0.342)      raw dot #1:  chunk  0  (11.48)
Q: How long are failed jobs kept in the dead-letter table?
   cosine #1:  chunk 19  (0.766)      raw dot #1:  chunk  0  (68.36)

Three of three, with the biggest scores on the page. Your retriever has become return longest_paragraph.

2. Ask something the corpus can't answer

argsort always returns k rows. There is no "I don't know" in top-k. Ask about a team that doesn't exist and you get the nearest team that does, with a better score than a real hit: 

Q (answer IS in corpus):  Which team owns Lantern?
* #1  0.342  chunk 12  Lantern is owned by the Platform team.

Q (answer NOT in corpus): Who is the tech lead of the Security team?
  #1  0.530  chunk 14  Priya Raman is the current tech lead of the Platform team.
  #2  0.201  chunk 12  Lantern is owned by the Platform team.
  #3  0.129  chunk 13  The Platform team's on-call rotation is one week long.

A model that skims will answer "Priya Raman". A score threshold would have to sit above 0.53 and below 0.34 at the same time. It can't. You need a reranker, or a model that actually reads the context and refuses — and an eval for it, because the retriever will never tell you.

3. Chunks too big: the fact is buried

Merge the 21 sentences into 3 chunks of 7. Same corpus, same questions, same maths. Every chunk now mentions Lantern, so every chunk looks a bit right, and the gap between hit and runner-up collapses: 

       #1 score  #2 score    gap   #1 is    answer in #1?
small chunks:
  Q1      0.720     0.315   0.405   chunk  4   yes
  Q2      0.342     0.174   0.168   chunk 12   yes
  Q3      0.766     0.398   0.368   chunk 19   yes

big chunks:
  Q1      0.535     0.232   0.303   chunk  0   yes, sentence 5 of 7
  Q2      0.215     0.206   0.009   chunk  2   NO
  Q3      0.226     0.200   0.026   chunk  0   NO

prompt words for Q1, k=3:  small chunks 103  big chunks 271  (whole document: 239)

Q2's gap goes from 0.168 to 0.009 — a coin toss — and the coin lands wrong. Q3 loses to the README paragraph, which has every one of its words, twice. And k=3 of 3 chunks is the whole document: the retriever did nothing, you pasted the wiki into the prompt and paid 271 words for a six-word question. Chunk size is the most consequential knob in the pipeline and the one nobody tunes.

There's a fourth, in demo 8. "Which port does the Lantern dashboard listen on?" needs two chunks, and the second one shares one word with the question and everything with the answer to hop one. At k=1 the prompt gets half a fact. The fix is to retrieve again after reading — which is what an agent loop is, and what RAG-Token gestures at.

The code

Pure NumPy. No torch, no model, no training — just the retriever, so you can watch the scores move. Every function under 15 lines.

Download the full script — nine short functions plus eight runnable demos. Then: 

python3 rag_from_scratch.py
D = 1024   # embedding width. DPR's is 768. Ours is a hash table, not a model.


def bucket(token):
    """Hashing trick: token -> a fixed slot in [0, D). No vocabulary to build,
    store or ship. crc32 rather than hash() because hash() is salted per
    process and the index would not survive a restart."""
    return zlib.crc32(token.encode()) % D


def fit_idf(chunks):
    """Inverse document frequency, one weight per bucket.

    A word in most chunks ("lantern") is worth little; a word in one chunk
    ("timeout") is worth a lot. log((N+1)/(df+1)) + 1, the smoothed textbook
    form. Same instinct as BM25, minus the tuning knobs."""
    df = np.zeros(D)
    for c in chunks:
        for b in set(bucket(t) for t in tokenise(c)):
            df[b] += 1
    return np.log((len(chunks) + 1) / (df + 1)) + 1


def embed(text, idf, normalise=True):
    """text -> (D,) vector.

    (1) count each bucket           -- term frequency
    (2) multiply by idf             -- rare words count more
    (3) divide by the L2 norm       -- so dot product == cosine, length-blind

    Step 3 is the one people delete. Demo 5 shows what happens."""
    v = np.zeros(D)
    for t in tokenise(text):
        v[bucket(t)] += 1
    v *= idf
    if normalise:
        v /= np.linalg.norm(v) + 1e-12
    return v


def retrieve(question, index, idf, k=3, normalise=True):
    """The whole retriever: one matrix-vector product and an argsort.

    scores[i] = similarity(question, chunk i). Return the k best, best first.
    There is no "nothing matched" -- argsort always returns k rows."""
    q = embed(question, idf, normalise)
    scores = index @ q                        # (n_chunks,)
    top = np.argsort(-scores)[:k]
    return [(int(i), float(scores[i])) for i in top]

chunk, tokenise, build_index (an np.stack) and build_prompt (the f-string you saw above) are in the script.

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living — reverse-engineering from broken behaviour is your native mode, and every row here is a bug you will one day ship.

Break thisWatch for
Pass normalise=False to build_index and retrieveChunk 0, the 37-token README paragraph, wins all three questions. Raw score 68.36 on a question it doesn't answer.
k=1 on the two-hop question in demo 8hop 2 in prompt: False. The prompt says "served by Beacon" and never mentions a port. At k=3 it arrives at rank 3, 0.187, next to the wrong port.
chunk(DOCUMENT, 7) — 3 giant chunks instead of 21Q2's gap drops from 0.168 to 0.009, and Q2 and Q3 both retrieve a chunk without the answer. The prompt grows from 103 words to 271.
build_prompt(q, []) — no retrievalContext: (none). The model answers from its weights. For a private corpus that's a guess dressed as a fact.
Ask "Who is the tech lead of the Security team?"Top hit 0.530 for the Platform team's lead — higher than a genuine hit at 0.342. No threshold separates them.
D = 32 instead of 1024The vocabulary line reports 87 hash collisions. "Which team owns Lantern?" now returns the max_workers sentence at 0.557 — two unrelated words share a slot.
Then read the paper — but only §2, the methods section, and the "Additional Results" part of §4: index hot-swapping, the retrieval ablations, the effect of retrieving more documents. Skip the FEVER and Jeopardy setup unless you need them. You'll find the paper is just the comments you've already read, with a BERT where my hash table is.

On the maths anxiety

The retriever is a dot product and an argsort. You've done both today. The tf-idf weight is a log, there for the usual reason: it squashes "in 1 chunk" versus "in 14" into a sensible range. L2 normalisation is a divide. That's the entire retrieval half.

The formula's Σ is a weighted average of k answers, and the weights are a softmax over k similarity scores — the same softmax as attention. The training loss is the negative log of that average: log so multiplication becomes addition and small probabilities don't underflow. loss.backward() does the rest, and it doesn't go through the index, because the paper froze it. You would too.

What you actually need for RAG work isn't maths: a chunking strategy, an embedding model, a reranker, an eval set of questions with known answers, and the habit of reading the retrieved passages before blaming the model.

Where to go next

This page gave you one thing: the retriever as a search engine you can break. That's deliberately narrow. These fill in the trained version, the family tree, and the real code.

Order I'd take them in: the paper's §2 → Weng → the Hugging Face docs. Method, then the neighbours, then the code. If you only read one, read §2 — it's shorter than this page.