LLaMA: Open and Efficient Foundation Language Models

Touvron et al., 2023

Not a new idea. A recipe. Explained as a diff, because that is what it is.

Download llama_from_scratch.py — everything below runs from it. NumPy only.

I'm an AI Engineer. I've served LLaMA-family models in production, quantised them, put them behind a load balancer, and I still couldn't have told you what was in one. I assumed there was a new mechanism. There isn't. LLaMA is the 2017 transformer with four config changes and a lot more data. The interview question is never "how does LLaMA work" — it's "what's different from the original transformer, and why". This is that answer.

LLaMA is a pull request against the 2017 transformer

You know what a diff looks like. Here is the whole architecture as one, against the decoder-only transformer — the GPT shape, causal mask and all: 

-norm:        layernorm, after the residual add
+norm:        rmsnorm, before each sublayer
-activation:  relu, hidden = 4d
+activation:  swiglu, hidden = 2/3 * 4d rounded up to a multiple of 256
-position:    sinusoid, added to the embeddings once at the bottom
+position:    rope, rotate Q and K inside every attention layer
-bias:        true
+bias:        false
 tokens:      1T to 1.4T, public data only     # not a hunk. The real change.

Four hunks. The attention itself is untouched — it's the lookup from the attention page. Three hunks are named in the paper's architecture section; the fourth (no bias) is only in the released code. And the line that isn't a hunk is the one that mattered.

TermMeansProgrammer's analogy
RMSNormRescale each token's vector so its root-mean-square is 1. No mean subtraction.Lower-casing before a string compare. Hygiene, not logic.
Pre-normThe norm sits on the branch, not the trunk: x + f(norm(x))Middleware that reads a normalised copy of the request and never rewrites the request.
SwiGLUFFN with a multiplicative gate: one branch scales the other.A feature flag that's a float, not a bool.
RoPERotate Q and K by their position before the dot product.Fold a timestamp into the hash so equal offsets compare equal.
1T tokensTrain 7× past the compute-optimal point.Precompute. Pay at build time so every request is cheaper.

The one formula

h = x + Attn(RMSNorm(x))
y = h + Wdown(SiLU(Wgate RMSNorm(h)) ⊙ Wup RMSNorm(h))

One block. Stack 32 for 7B, 80 for 65B. Read it as pipeline stages:

StepShapeWhat it does
RMSNorm(x)(n, d)Every token onto the same scale. Reads the trunk, doesn't write it.
Attn(·)(n, d)The lookup from attention, RoPE applied to Q and K inside
x + …(n, d)The skip link. Adds. Never rescales. This is the pre-norm hunk.
Wgatez, Wupz(n, h), h = 11008Two branches off the same normalised input
SiLU(·) ⊙ ·(n, h)The gate. One branch scales the other, elementwise
Wdown(n, d)Back to trunk width, added to the trunk. Shape in equals shape out, so it stacks
after block 32(n, 32000)One final RMSNorm, then the output head. No bias anywhere

RMSNorm: LayerNorm minus the mean

LayerNorm subtracts the mean, divides by the standard deviation, then a learned scale γ and shift β. RMSNorm (Zhang and Sennrich, 2019) skips the mean and the shift: divide by the root-mean-square, multiply by γ. Same job — every token onto the same scale before it hits a matmul — with fewer passes over the vector, and no quality loss they could measure.

Don't take my word for it. Here's the actual output:

x           = [1. 2. 3. 4.]   mean 2.5000
layer_norm  = [-1.3416 -0.4472  0.4472  1.3416]   mean 0.0000
rms_norm    = [0.3651 0.7303 1.0954 1.4606]   mean 0.9129
same numbers? False   ← the mean is still in there

x centred   = [-1.5 -0.5  0.5  1.5]   mean 0.0000
layer_norm  = [-1.3416 -0.4472  0.4472  1.3416]
rms_norm    = [-1.3416 -0.4472  0.4472  1.3416]
same numbers? True    ← identical once centred

On a centred vector they are the same function; real activations aren't centred, so the outputs differ by exactly the mean. What you buy, counted off my implementation — seven passes over the vector versus four: 

elementwise ops per token per norm at d=4096:
  layer_norm 28,672   rms_norm 16,384   saved 12,288
  x 2 norms per layer x 32 layers = 786,432 ops saved per token

rms_norm(zeros, eps=0)    = [nan nan nan nan]
rms_norm(zeros, eps=1e-5) = [0. 0. 0. 0.] ✓

Nothing next to a 4096 × 11008 matmul, but it's paid on every token, on every layer, for free. And eps is not decoration: 0/0 is NaN, and one NaN in the residual stream poisons every layer above it.

Pre-norm: the norm goes on the branch, not the trunk

2017: x = norm(x + f(x)). The norm sits on the trunk; every block's output is renormalised before the next block sees it. LLaMA: x = x + f(norm(x)). The norm sits on the branch; the trunk is a straight wire from embedding to output, and every block only ever adds to it. You will read that pre-norm "keeps the activations bounded". Run demo 3. It doesn't — post-norm does. What pre-norm keeps bounded is something else: 

d=64, sqrt(d)=8.0. residual-stream norm per token:
                     layer 1   layer 5   layer 10  layer 20   bump at 0 -> out
  pre-norm  (LLaMA)      10.8      18.9      27.7      40.7   x 1.77
  post-norm (2017)          8         8         8         8   x 9.71
  no norm at all           12  6.23e+05       inf       nan   x nan

Post-norm pins the trunk at √d = 8 exactly. Pre-norm lets it grow, and nobody cares, because no sublayer ever reads x, only rms_norm(x). Delete the norm and the trunk overflows before layer 10 — which is why there's a norm in there at all.

The last column is the one that matters. It's a finite-difference gradient: bump the input by a tiny δ, measure how far the output moves, divide — the trick you'd use to unit-test a hand-written backward pass. It's how much a change at layer 0 is amplified by the top, which is also how much a gradient at the top is amplified on its way down to layer 0. Now go deeper: 

same bump at layer 0, deeper and deeper stacks:
                      5 deep   10 deep   20 deep   32 deep
  pre-norm  (LLaMA)  x   1.47  x   1.58  x   1.77  x   1.68
  post-norm (2017)   x   1.69  x   2.74  x   9.71  x   40.6   ← compounds
Post-norm multiplies per-layer factors down the stack; pre-norm adds them. A gradient ratio that grows with depth is a training run that needs warmup, a small learning rate and luck. That's the stability argument, and why GPT-2 onwards, PaLM and LLaMA are all pre-norm. Xiong et al., 2020 have the proof; Liu et al., 2020 call it the amplification effect. The exact ratios depend on the init scale; the compounding doesn't.

SwiGLU: a gate in the FFN, paid for with a narrower hidden layer

The 2017 FFN is relu(x @ W1) @ W2, hidden width 4d. SwiGLU (Shazeer, 2020) replaces it with (silu(x @ W_gate) * (x @ W_up)) @ W_down. Two branches see the same input; one is squashed through SiLU and multiplies the other, elementwise. A gate. Three matrices instead of two, so to keep the parameter count level the hidden width is cut to two-thirds, then rounded up to a multiple of 256 because GPUs like that. Hence 11008, not 10923. 

d = 4096
  vanilla ReLU FFN:  hidden = 4d   = 16384   params = 2*d*h = 134,217,728
  LLaMA SwiGLU FFN:  hidden = 2/3*4d = 11008   params = 3*d*h = 135,266,304
  ratio SwiGLU / ReLU = 1.008

BREAK: SwiGLU with hidden = 4d = 16384:   params = 201,326,592   (+50.0%)

Within 1%. That's the paper's point: the comparison is parameter-matched, so any benchmark gain is the gate, not extra capacity. Leave the hidden at 4d and you've grown the FFN by half and proved nothing. Demo 2 also runs a toy forward pass: ReLU switches 49.0% of hidden units exactly off, SwiGLU 0.0% — a switch versus a dimmer. Why does the dimmer help? Nobody knows. Shazeer's paper says so: "we attribute their success, as all else, to divine benevolence". Don't let an interviewer tell you there's a derivation.

RoPE: rotate Q and K, leave V alone

2017 added a position vector to the embeddings once, at the bottom. LLaMA adds nothing there. Inside every attention layer it rotates each query and key by an angle proportional to its position (Su et al., 2021). Rotate both by the same rule and the dot product depends only on the gap: 

q at 3 . k at 1  (offset 2) = 1.7986
q at 7 . k at 5  (offset 2) = 1.7986   ← same offset, same score
q at 3 . k at 0  (offset 3) = 1.7523   ← different offset

where it is applied, straight from attn_sublayer's source:
  return attention(rope(Q, pos), rope(K, pos), V) @ p["Wo"]  # rope: Q, K. Not V

It goes on Q and K because those are the two things being dotted. V is the content handed over once a match is made; rotate that and the content would depend on where the token sits. The pairs, the frequencies, why it extrapolates: the RoPE page. Here the point is where it's called.

6.7B parameters, 1T tokens: the arithmetic

Every LLaMA-1 size is one formula with four numbers from the paper's Table 2. Attention is 4d² (four d × d matrices — no GQA yet, that's LLaMA 2 at 70B), the FFN is 3dh, two norms per layer, an embedding table, an untied output head: 

size    dim  layers     ffn      count_params   paper
  7B   4096      32   11008     6,738,415,616    6.7B ✓
 13B   5120      40   13824    13,015,864,320   13.0B
 33B   6656      60   17920    32,528,943,616   32.5B
 65B   8192      80   22016    65,285,660,672   65.2B

BREAK: put every bias term back (bias=True):
  6,740,037,632 vs 6,738,415,616: +1,622,016 = +0.024%

Within rounding of every row in the paper. The FFN is twice the attention — which is why mixture-of-experts goes after the FFN. Dropping the biases was a simplicity decision, not a savings one: 0.024%. The paper doesn't mention it. Now the tokens. Chinchilla (Hoffmann et al., 2022) answered one question: for a fixed training budget, what model size and how many tokens minimise the loss? About 20 tokens per parameter, so a 7B model "should" see 135B tokens. LLaMA-7B saw a trillion. 

LLaMA-7B parameters                 6.74e+09
Chinchilla-optimal tokens, 20/param 1.35e+11  (135B)
LLaMA-7B actually trained on        1e+12  (1T)   7.4x more

inference compute ~ 2 * N per token:
  LLaMA-7B     1.35e+10 FLOPs/token
  GPT-3 175B   3.5e+11 FLOPs/token   26x more

tokens per parameter:
  GPT-3 175B 300B / 175.0B =   1.7
  LLaMA-7B   1.0T /  6.7B = 148.4
  LLaMA-13B  1.0T / 13.0B =  76.8

The argument is a cost model you already run in your head. Training is paid once. Inference is paid per token, forever, and scales with parameter count: 7B costs 26× less per token than 175B. Serving a billion requests, you'd rather overspend training a small model than underspend and serve a big one. The paper's training curves show the 7B loss still falling at a trillion tokens — slower than at 135B, but falling. And that, not the four hunks, is why LLaMA-13B beat GPT-3 175B on most benchmarks in the paper: GPT-3 saw 1.7 tokens per parameter, LLaMA-13B saw 77.

If you're asked "why did LLaMA train past compute-optimal": Chinchilla optimises the training bill; LLaMA optimises the serving bill. That's the "efficient" in the title. 1T tokens for 7B and 13B, 1.4T for 33B and 65B, public data only; 65B took 21 days on 2048 A100s.

The code

Pure NumPy. No torch, no autograd, no training — forward passes and parameter arithmetic, so you can watch the numbers move. Every function under 15 lines. The four deltas below; count_params, the post-norm block and the finite-difference probe are in the script.

Download the full script — 130 lines plus six runnable demos. Then: 

python3 llama_from_scratch.py
import numpy as np


def rms_norm(x, eps=1e-5):
    rms = np.sqrt((x ** 2).mean(axis=-1, keepdims=True) + eps)
    return x / rms


def silu(x):
    return x / (1.0 + np.exp(-x))                    # x * sigmoid(x)


def llama_hidden(dim, multiple_of=256):
    hidden = int(2 * (4 * dim) / 3)                  # 2/3 of the vanilla 4d
    return multiple_of * ((hidden + multiple_of - 1) // multiple_of)


def swiglu_ffn(x, W_gate, W_up, W_down):
    return (silu(x @ W_gate) * (x @ W_up)) @ W_down  # (n,d)->(n,h)->(n,d)


def rope(x, positions, base=10000.0):
    d = x.shape[-1]
    theta = base ** (-np.arange(0, d, 2) / d)        # (d/2,) one per pair
    angle = positions[:, None] * theta[None, :]      # (n, d/2)
    cos, sin = np.cos(angle), np.sin(angle)
    x1, x2 = x[..., 0::2], x[..., 1::2]              # even dims, odd dims
    out = np.empty_like(x)
    out[..., 0::2] = x1 * cos - x2 * sin             # a 2-D rotation per pair
    out[..., 1::2] = x1 * sin + x2 * cos
    return out


def pre_norm_block(x, p):                            # LLaMA
    x = x + attn_sublayer(rms_norm(x), p)
    x = x + ffn_sublayer(rms_norm(x), p)
    return x

Now break it

This is the part that makes it stick. Reading won't do it. You debug for a living — reverse-engineering from broken behaviour is your native mode.

Break thisWatch for
In llama_hidden, return 4 * dimThe 7B FFN goes from 135,266,304 to 201,326,592 parameters, +50.0%. The "SwiGLU is better" claim stops being parameter-matched and means nothing.
rms_norm(np.zeros(4), eps=0.0)[nan nan nan nan]. 0/0. One NaN in the residual stream poisons every layer above it. With eps=1e-5: zeros.
In demo_3, run the post-norm ordering on blocks[:32] instead of [:20]The bump-at-layer-0 ratio goes 9.71× → 40.6×. Pre-norm on the same 32 blocks: 1.77× → 1.68×. One compounds with depth, the other doesn't.
Delete both rms_norm calls in pre_norm_block (that's no_norm_block)Trunk norm 12 → 6.23e+05 by layer 5 → inf by layer 10 → nan. That's why there is a norm at all.
count_params(4096, 32, 11008, bias=True)+1,622,016 parameters, +0.024%. Dropping bias was for a simpler model.py, not for savings.
Set tokens=1.35e11 in LLAMA["7B"] (Chinchilla-optimal)The "7.4x more" lines print 1.0x. Training cost drops 7.4×; nothing else in a forward pass changes. What you'd lose — the loss curve past 135B tokens — is in the paper's training-loss figure, not in any script. Cheaper to train, worse to serve for the same quality.
Then read the paper — but only §2, the approach: the data mix, the architecture section (half a page; it is this diff), the optimiser paragraph, Table 2, and the training-loss figure. Skip the benchmark tables in §3 and the bias, toxicity and carbon sections unless you need them. You'll find the paper is just the comments you've already read.

On the maths anxiety

This paper needs less maths than the attention one. A mean, a root-mean-square, a 2-D rotation (sine and cosine from school), and multiplying four numbers to count parameters. The scaling-law part is one rule of thumb — training FLOPs ≈ 6 × parameters × tokens, inference ≈ 2 × parameters per token — a cost estimate, not a theorem. You don't need to derive why SwiGLU beats ReLU; nobody has. You don't need the proof that pre-norm gradients behave; you need the finite-difference trick above: "does this multiply or add down the stack".

What separates AI engineers on this paper is the arithmetic: count 6.7B on a napkin, say why 13B is cheaper to serve than 175B, say what 20 tokens per parameter means for your training budget. None of it is research maths.

Where to go next

This page gave you LLaMA as a diff, with the four hunks measured. It skipped the tokeniser, the data pipeline, the optimiser, and running real weights.

Order I'd take them in: the paper's §2 → Raschka → model.py. Text, then the diff applied live, then the production version. If you only read one, read model.py — it's what you'll be serving. For the forward pass with nothing to hide behind, Karpathy's llama2.c: one C file, rmsnorm as a for loop.