Parameter-Efficient Fine-Tuning: the family

Houlsby et al., 2019 — with Li & Liang 2021, Lester et al. 2021, Liu et al. 2022

Five papers, one question: where do you plug the small trainable thing into the big frozen one?

Download peft_from_scratch.py — everything below runs from it. NumPy only.

I'm an AI Engineer. I've run get_peft_model() in production, picked r=8 because the tutorial did, and I still couldn't tell you why an adapter adds latency and a LoRA doesn't. Five papers, five acronyms, and nobody told me they are one paper asked five ways.

PEFT is middleware

You know this. The framework handles the request. You don't fork it:

def handle(request):              # the framework. Frozen. Not yours.
    return framework.handle(request)

app.use(my_middleware)            # small, yours, sits in the request path

Full fine-tuning is forking the framework: every file is yours now, one copy per task. PEFT is registering middleware. Freeze the base, put one small trainable thing in the path, train only that: 

for p in model.parameters():
    p.requires_grad = False        # the whole method is this line

def adapter(h):                    # Houlsby 2019. h flows through untouched
    return h + W_up @ relu(W_down @ h)   # plus a tiny, learned diff

Every method on this page is that, with the hook somewhere else.

TermMeansProgrammer's analogy
Frozen baseThe pretrained weights. Never updatedThe framework you pip install
HookThe few parameters that do trainYour middleware
Residual h + f(h)The hook proposes a diff, not a replacementA skip link. next() still runs
BottleneckSqueeze d → m → d with m ≪ dA thin interface, not a second framework
MergeFold the hook back into the frozen weightsInlining. Zero cost at runtime

Did it beat full fine-tuning? Not on accuracy — on everything else. Full fine-tuning BERT-large moves 340M parameters and stores a 340M checkpoint per task. Houlsby's adapters train about 3.6% of the parameters and land within 0.4 points of full fine-tuning on GLUE. So the interview question is never "which is more accurate". It's "where does each go, what does it cost to train, what does it cost to serve".

The one formula

Adapter(h) = h + Wup · ReLU(Wdown · h + bdown) + bup

Read it inside-out, as pipeline stages:

StepShapeWhat it does
h(n, d)The sublayer's output. n tokens, d wide. Frozen code made it
Wdown(d, m)Squeeze. d=64 down to m=8. This is where the saving lives
ReLU(n, m)A nonlinearity, so the two matrices don't collapse into one
Wup(m, d)Unsqueeze. Back to d. Initialised to zero
h +(n, d)Skip link. Shape unchanged, so it drops into any block

Two per block in the paper: after attention and after the feed-forward, before each add-and-norm. Houlsby also unfreezes the layer norms; I leave those frozen to keep the count clean.

Five hooks, one frozen block

The script builds one block — d=64, one head, FFN 4d — as a dict of arrays. Nothing in it trains. Each method is a function that returns only its trainable thing, and n_params() counts it. 

  attention (W_q, W_k, W_v, W_o):   16,384
  feed-forward (W_1, b_1, W_2, b_2):  33,088
  two layer norms:                      256
  TOTAL frozen params:               49,728

method                   per block  % block  x12 blocks
full fine-tuning            49,728  100.00%      49,728
adapters, m=8 (x2)           2,192    4.41%      26,304
prefix tuning, p=8           1,024    2.06%      12,288
prompt tuning, p=8             512    1.03%         512
(IA)^3                         384    0.77%       4,608
LoRA r=8 on W_q, W_v         2,048    4.12%      24,576

Every method except prompt tuning pays per layer; prompt tuning pays once, at the input, however deep the model is. The percentages are high only because the toy is tiny — a hook grows with d, a block with d² — which is how these become Houlsby's 3.6% and (IA)³'s 0.01% at real widths. And this is the table you'll be asked for:

MethodWhere the hook goesTrainable (toy)Inference cost
Adapter Houlsby 2019After each sublayer, inside the residual2,192 (4.41%)Extra sequential matmuls. Latency
Prefix tuning Li & Liang 2021p extra K and V rows, every layer1,024 (2.06%)+p entries in every layer's KV cache
Prompt tuning Lester 2021p extra input rows, once512 (1.03%)+p tokens through the whole stack
(IA)³ Liu 2022Scaling vectors on K, V and the FFN hidden384 (0.77%)Zero, after merging
LoRA Hu 2021Low-rank diff on Wq, Wv2,048 (4.12%)Zero, after merging

Step 0 must be a no-op

A hook that changes the output before training has thrown the pretrained model away on the first forward pass. Houlsby's fix: zero the up-projection. Which of the five start clean? 

max |block_with_hook(X) - block(X)| before any training:
  adapters, W_up = 0:           0.000000
  adapters, W_up random:        0.376937  ← not a no-op
  (IA)^3, vectors = 1:          0.000000
  prefix, K and V = 0:          0.277744  ← not a no-op
  prompt, 8 sampled tokens:     0.491051  ← not a no-op

Adapter and (IA)³: byte-identical. The prefix is not, even zeroed — a zero key still scores exp(0) = 1 in the softmax, so it takes attention mass from real tokens before anything is learned. Prompt tokens likewise: they are seen. LoRA starts clean for the adapter's reason: B = 0.

Zero the last matrix, not the first. Zero Wdown instead and the gradient into Wup is zero too; training never leaves the origin. Same trick in LoRA.

Prefix tuning: keys decide, values deliver

Li and Liang moved the hook into attention. Prepend p learned vectors to the keys and p to the values, every layer, so real tokens' queries have p extra slots to look at. Nothing passes through Wk or Wv: the prefix is the keys and values. The paper trains about 0.1% of the parameters this way on GPT-2 and BART (via a small MLP for stability, thrown away after; I skip it). Now learn only the values: 

attention weights shape with p=4 prefix rows: (16, 20)  (n, p+n)
weight each query puts on the 4 prefix slots (first 3 queries):
[[0.043 0.027 0.05  0.047]
 [0.071 0.061 0.017 0.067]
 [0.083 0.04  0.023 0.007]]

BREAK IT: train V only, leave the prefix keys at zero
[[0.035 0.035 0.035 0.035]
 [0.027 0.027 0.027 0.027]
 [0.032 0.032 0.032 0.032]]
change every value row, max |weights diff|: 0.0

With learned keys each query reads the four slots differently. With zero keys the columns are identical and redrawing every value vector moves the weights by exactly 0: the prefix has collapsed into one fixed vector added to everyone. Keys are the addressing, values the payload. You need both.

Prompt tuning: the hook is in the input

Lester et al. asked the smallest version of the question: what if the only trainable thing is p token embeddings, prepended once at the input? No hook in any block; they are ordinary positions and every layer sees them. 

input: 8 prompt rows + 16 real rows = (24, 64)
attention mass real tokens put on the prompt:
  mean 0.340   min 0.196   max 0.546   (uniform would be 0.333)

BREAK IT: read the output at row 0
  out.shape = (24, 64)   ← 8 of those rows are the prompt
  out[0]  belongs to prompt[0]      out[8] belongs to real token 0
  the answer is out[8:], shape (16, 64)

A third of the attention mass, on tokens nobody has trained yet (sampled from real token vectors, the paper's recommended init). Training moves them so what real tokens read there is useful. And the output has 24 rows: out[0] is a prompt token's output. Slice the prompt off.

The honest caveat. Lester's headline finding is that prompt tuning only matches full fine-tuning once the model passes roughly 10B parameters. Below that it lags. I can't show that without training, so I'm telling you instead: if your model is BERT-sized, wrong hook.

(IA)³ and LoRA: hooks that fold away

Liu et al. went smaller still. (IA)³ learns three vectors per block — one scales the keys, one the values, one the FFN hidden — and nothing else. 384 numbers here, about 0.01% of a real model. A scale is a multiply, so afterwards you fold it into the frozen matrix (columns of Wk and Wv, rows of W2) and the hook is gone. That's the column the family is really sorted by: 

base block FLOPs: 1,638,400

method               extra FLOPs  extra tokens  adds a step?
adapters, m=8             65,536             0           yes
prefix, p=8               32,768             8            no
prompt, p=8              868,352             8            no
(IA)^3, unmerged           6,144             0            no
(IA)^3, merged                 0             0            no
LoRA, merged                   0             0            no

(IA)^3 merge check, max |hooked - merged|: 1.33e-15   identical? True

The adapter's 65,536 extra FLOPs are 4% of the block. Small. But they sit in the residual path: the block can't finish until the adapter does, a serial delay paid every layer, every token. That was Houlsby's cost, and the gap LoRA closed: put the diff on the weight matrix itself, W + BA, and merge it after training. Same idea, zero serving cost. It has its own page; I won't re-explain it here.

Prefix and prompt tuning don't add a step, they widen one. But the prompt row is 868,352 extra FLOPs, half the block again: 8 more tokens through every matmul in every layer. Nothing is free; pick where.

The code

Pure NumPy. No torch, no autograd, no training — the forward pass with every hook point marked, so you can put a breakpoint in block() and watch which one fires. Every function under 20 lines.

Download the full script — 170 lines plus six runnable demos. Then: 

python3 peft_from_scratch.py
def adapter(h, A):
    z = relu(h @ A["W_down"] + A["b_down"])          # (n, d) -> (n, m)
    return h + z @ A["W_up"] + A["b_up"]             # (n, m) -> (n, d), + skip


def block(W, X, hooks=None, return_weights=False):
    h = hooks or {}
    Q, K, V = X @ W["W_q"], X @ W["W_k"], X @ W["W_v"]
    if "ia3_k" in h:
        K, V = K * h["ia3_k"], V * h["ia3_v"]          # (IA)^3: scale K, V
    if "prefix_k" in h:                                # prefix: p extra rows
        K = np.concatenate([h["prefix_k"], K])         #   the queries can
        V = np.concatenate([h["prefix_v"], V])         #   attend to
    weights = softmax(Q @ K.T / np.sqrt(Q.shape[-1]))  # (n, n + p)
    a = (weights @ V) @ W["W_o"]
    if "adapter_attn" in h:
        a = adapter(a, h["adapter_attn"])              # Houlsby hook 1
    X = layernorm(X + a, W["ln1_g"], W["ln1_b"])
    f = relu(X @ W["W_1"] + W["b_1"])
    if "ia3_ff" in h:
        f = f * h["ia3_ff"]                            # (IA)^3: scale hidden
    f = f @ W["W_2"] + W["b_2"]
    if "adapter_ffn" in h:
        f = adapter(f, h["adapter_ffn"])               # Houlsby hook 2
    X = layernorm(X + f, W["ln2_g"], W["ln2_b"])
    return (X, weights) if return_weights else X


def make_adapters(m, d=D, zero_up=True, seed=1):
    """Houlsby: two bottleneck adapters per block, width m. Up-proj zeroed so
    step 0 is the identity (the paper: 'near-identity initialisation')."""
    rng = np.random.default_rng(seed)
    def one():
        up = np.zeros((m, d)) if zero_up else rng.normal(0, 0.1, (m, d))
        return {"W_down": rng.normal(0, 0.1, (d, m)), "b_down": np.zeros(m),
                "W_up": up, "b_up": np.zeros(d)}
    return {"adapter_attn": one(), "adapter_ffn": one()}

def make_prefix(p, d=D, zero=False, seed=2):
    """Li & Liang: p learned key rows and p learned value rows, per layer.
    They never pass through W_k / W_v -- they ARE the keys and values."""
    rng = np.random.default_rng(seed)
    if zero:
        return {"prefix_k": np.zeros((p, d)), "prefix_v": np.zeros((p, d))}
    return {"prefix_k": rng.normal(0, 1, (p, d)),
            "prefix_v": rng.normal(0, 1, (p, d))}

def make_prompt(p, X, d=D, seed=3):
    """Lester: p learned INPUT tokens. Not a hook in the block -- new rows
    prepended to X. Init by sampling real token vectors (the paper's best)."""
    rng = np.random.default_rng(seed)
    return X[rng.integers(0, len(X), size=p)].copy()

def make_ia3(d=D, d_ff=D_FF):
    """Liu 2022: three scaling vectors, all ones at init -> identity."""
    return {"ia3_k": np.ones(d), "ia3_v": np.ones(d), "ia3_ff": np.ones(d_ff)}

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living; reverse-engineering from broken behaviour is your native mode.

Break thisWatch for
In demo 2, change make_adapters(8) to make_adapters(64) — bottleneck = d16,640 trainable, 33.46% of the block: more than the whole attention sublayer (16,384) you bolted them onto. No bottleneck, no method.
In demo 3, pass zero_up=False to make_adaptersStep-0 drift goes from 0.000000 to 0.376937. You overwrote the pretrained behaviour before the first gradient.
In demo 4, keep "prefix_k": np.zeros((p, D)) and only redraw prefix_vEvery prefix column is identical per row (0.035 0.035 0.035 0.035) and the weights move by 0.0. The prefix can't be addressed.
In demo 5, read out[0] instead of out[8:]Your "answer" is prompt token 0's output. out.shape is (24, 64); the real tokens are the last 16 rows.
Count n_params(hooks) + n_params(W) — what forgetting requires_grad = False does in a framework51,920 trainable, 104.41% of the block. Full fine-tuning with extra steps, and the checkpoint is the whole model again.
Then read the papers — method sections only. Houlsby: section 2 and its Figure 2, then the GLUE table. Lester: the figure that plots prompt tuning against model size. Each is one row of the hook table.

On the maths anxiety

There's less here than on the attention page. The family is matrix shapes and one skip link. An adapter is two matmuls and an add. A prefix is a concatenate; a prompt is a concatenate one call earlier. (IA)³ is a multiply. Gradients are still loss.backward().

What separates AI engineers here is knowing which hook merges, what each does to the KV cache, and that prompt tuning's scale threshold is real.

Where to go next

This page gave you the family as one hook table, in code you can break. These cover what I skipped: the history and the pictures.

Order I'd take them in: Raschka → Houlsby → soft prompts. History, anchor paper, then the branch I treated briefly. If you only read one, read Houlsby section 2 — it's adapter() with a picture.