LoRA: Low-Rank Adaptation of Large Language Models

Hu et al., 2021

Explained as a git patch, because a fine-tune is a diff and this paper is about how small the diff is.

Download lora_from_scratch.py — everything below runs from it. NumPy only.

I'd shipped LoRA adapters before I understood them. r=8, lora_alpha=16, target_modules=["q_proj", "v_proj"], copied from a blog post, and it worked. I couldn't have told you what r was, why alpha existed, or why the README promised zero inference latency. Nobody told me the whole paper is one observation about a diff.

A fine-tune is a patch, not a fork

You know this. A full fine-tune is a fork: copy every weight, edit every weight, keep the copy.

W_task = W_base.copy()       # 4096 x 4096 floats, per matrix, per task
W_task += learned_change     # every entry is allowed to move
save(W_task)                 # another full copy on disk

LoRA is the branch that stores only the patch. W_base stays read-only; the change lives next to it. And the paper's observation is that the patch is thin: rank r, where r can be 8, or even 1, on a matrix 4096 wide. 

A = random(d_in, r)          # r = 8, not 4096
B = zeros(r, d_out)          # the patch starts empty
h = x @ W_base + (x @ A) @ B * (alpha / r)

W_merged = W_base + A @ B * (alpha / r)     # ship time: apply the patch

Two thin matrices instead of one fat one. At ship time, apply the patch and serve one matrix, as before. That's the paper.

TermMeansGit analogy
W0The pretrained weight. Frozen.main, read-only
ΔWWhat a fine-tune changesThe diff
rRank: how many independent directions the diff may haveHow thin the patch is
B, AThe diff stored as two thin matrices, ΔW = BAThe patch, stored compressed
αA constant. The patch is scaled by α/rA learning-rate knob for the patch

Why this beat adapters and full fine-tuning

The setting is GPT-3, 175 billion parameters. A full fine-tune per task is a full copy of the weights per task — the paper puts one checkpoint at 350GB — plus gradients and optimiser state for all of them while you train. Adapters were the existing fix: a small two-layer MLP bolted after each layer. But that is a new layer in the sequential path with a nonlinearity inside, so it can't be folded away and it costs latency on every request. The paper measures this in its "aren't existing solutions good enough?" section: noticeable at batch size 1, where online inference lives.

LoRA's patch is linear, so it folds into the weight. Same latency as the base. Switching tasks is subtract one patch, add another. The checkpoint per task is the patch alone: the paper reports 35MB where a full fine-tune was 350GB.

The one formula

h = W0x + ΔWx = W0x + BAx     ΔW = BA,  B is d×r,  A is r×k,  r ≪ min(d, k)

Read it as pipeline stages:

StepShapeWhat it does
W0x(d,)The frozen base. What the model already does.
Ax(r,)Squeeze x down to r numbers. Random Gaussian at init.
B(Ax)(d,)Expand back out. Zero at init, so the patch starts empty.
× α/rscalarKeeps the step size fixed when you change r. Not in equation 3; in the paragraph after it.
W0 + BA(d, k)The merge. Done once before serving. One matrix again.

Two notes. The paper writes BA with column vectors; my x is a row, so the patch reads (x @ A) @ B, A then B, and my A @ B is the paper's BA transposed. And "rank": a rank-1 matrix is one column times one row, d×k entries but d+k numbers of information; rank r is r of those added up. np.linalg.svd lists a matrix's singular values, biggest direction first.

How many numbers are you training?

A full fine-tune of one d×d matrix trains d² numbers. LoRA trains r·(d + d). The paper's GPT-3 numbers fall out of the same formula: 96 layers, dmodel 12288.

Don't take my word for it. Here's the actual output:

one square weight, d = 4096. Full fine-tune trains d*d = 16,777,216

    r   LoRA params   full / LoRA
    1         8,192        2,048x
    4        32,768          512x
    8        65,536          256x
   64       524,288           32x

GPT-3 175B shape: 96 layers, d_model 12288. LoRA on Wq and Wv only:
  r = 1:     4,718,592 trainable =   4.7M   (175,000M full, ratio 37,087x)
  r = 8:    37,748,736 trainable =  37.7M   (175,000M full, ratio 4,636x)

4.7M and 37.7M are the two LoRA rows in the paper's GPT-3 results table; the abstract's "10,000 times" is this ratio, rounded. Note what they adapted: Wq and Wv in attention, nothing else; the MLP was frozen. Now the thing the paper is honest about and blog posts are not: LoRA shrinks what you train, not what you load. 

What LoRA does NOT shrink (d = 4096, r = 8, fp16 weights, fp32 Adam):
  the frozen W itself ............   33.6 MB  either way
  full FT  grads + Adam m, v .....  167.8 MB
  LoRA     grads + Adam m, v .....    0.7 MB
LoRA cuts gradients, optimiser state and checkpoints. The base costs the same memory at inference. That is why the paper's GPU memory saving is about 3×, not 10,000×. Shrinking the base is quantisation's job — QLoRA is "4-bit base plus a LoRA patch".

The diff is low-rank — and here's what that means

I can't fine-tune GPT-3 on a laptop, so here is the honest version for one linear layer. Least squares gives the ΔW a full fine-tune would converge to — closed form, no training loop — and an SVD says how many directions it has. The task's true ΔW is rank 4 plus a little noise, the paper's claim about real diffs. d = 64, 512 rows. 

  singular values of the fitted dW (first 10):
   [3.001 2.    0.986 0.512 0.156 0.151 0.146 0.14  0.137 0.135]

  rank r   dW error   test RMSE
    none                 0.4852   ← the base model
       1     0.6199      0.3026
       2     0.3330      0.1694
       4     0.1628      0.0907   ← the cliff
       8     0.1432      0.0840
      64     0.0000      0.0533   ← full fine-tune

Four real singular values — 3, 2, 1, 0.5 — then a noise floor at 0.15. The fitted ΔW has 4096 entries and 4 directions that matter. Rank 4 takes the test error from 0.49 to 0.09; the full fine-tune gets 0.05, and the gap is 60 tiny noise directions rank 8 can't reach and doesn't need. Now break it: same size of ΔW, random Gaussian. 

BREAK IT: same size of dW, but full-rank (random Gaussian):
   [0.928 0.878 0.847 0.828 0.803 0.793 0.774 0.75  0.737 0.719]   ← no cliff

  rank r   dW error   test RMSE
       8     0.7909      0.3828   ← rank 8 barely helps
      64     0.0000      0.0536   ← full fine-tune

No cliff, no LoRA: rank 8 leaves 79% of the diff on the floor. LoRA works because real fine-tune diffs look like the first table — an empirical claim, not a theorem. The paper's evidence is section 7: on a fine-tuned GPT-3 the top directions of an r = 64 update overlap heavily with the r = 8 one, and r = 1 to 4 already matched full fine-tuning on the tasks tested. The best part of the paper; everyone skips it.

Merge it and pay nothing at inference

x @ (W + AB) and x @ W + (x @ A) @ B are the same number — distributivity. So before serving, add the patch into the weight once. The served model is one matmul, the same one it always was. 

max |unmerged - merged| = 3.55e-15
flops per token, base W alone .........  8,192
flops per token, base + unmerged patch  10,240
flops per token, merged ...............  8,192   ← same as base
un-merge, max |W_back - W| = 5.55e-17   ← base recovered

best single matrix that reproduces a LoRA layer:  residual 1.83e-15
best single matrix that reproduces an adapter:    residual 5.44e-01

The last two lines are the whole argument against adapters. I asked least squares for the one matrix that best reproduces each layer. For LoRA it finds one exactly. For an adapter — two matmuls with a relu between — the best single matrix is off by 54%, because no linear map reproduces a relu. So it stays as extra layers, and extra layers are extra latency.

The interview question is "why does LoRA add no inference latency?" One line: the patch is linear, so it merges into W. Bonus: un-merge is subtraction.

Why B starts at zero (and not A)

With B = 0 the patch is zero and step 0 of your fine-tune is bit-for-bit the base model. Not close — identical. It can only move away from something that works. 

A random, B zero:    identical to base? True   max |diff| = 0.0
A random, B random:  identical to base? False   outputs moved 31.5%, untrained

  init                  |dL/dA|  |dL/dB|   what happens
  A random, B zero        0.000   11.009   B moves first, then A. Trains.
  A zero,   B zero        0.000    0.000   both zero forever. Stuck.
  A random, B random     10.724   11.009   trains, but step 0 is wrong.

Why zero B rather than A, or both? The second table is one gradient, by hand. dL/dA is multiplied by B, so with B = 0 A sits still for one step — but B gets a real gradient, and once B moves, A follows. Zero both and every gradient is zero; the patch never leaves the origin. Random both and you broke the model before training started.

What α is for

The paper scales the patch by α/r and says: set α to the first r you try, then leave it. Why that works: Adam's first step is exactly lr × sign(grad), the same size for every parameter. B has r rows and the patch sums over all of them, so a bigger r means a bigger step for the same learning rate — unless you divide by r. 

one Adam step (lr = 0.001) on B, from B = 0. alpha = 4.

    r   no scale: |dh|  loss drop   alpha/r: |dh|  loss drop
    1             0.1031    0.00014           0.4125    0.00054
    4             0.2553    0.00063           0.2553    0.00063
   16             0.5219    0.00246           0.1305    0.00062
   64             1.3644    0.01066           0.0853    0.00067

loss drop at r = 64 vs r = 1:  no scale 78x   alpha/r 1.2x

Without the scale, going from r = 1 to 64 makes the same optimiser step do 78× more to the loss: you silently changed your learning rate. With α/r the loss drop holds within 20%. (The r = 4 row is the sanity check: α/r = 1 there, so both halves match.) One honest caveat: under α/r the norm column still shrinks with r; it is the loss change that holds, and a 2023 follow-up, rsLoRA, argues for α/√r on those grounds. In practice α is a learning-rate knob for the patch — that is all the common "α = 2r" rule of thumb is.

The code

Pure NumPy. No torch, no autograd, no training loop — the fine-tune is a least-squares solve and the low-rank part is an SVD, so you can watch the diff get compressed.

Download the full script — about 100 lines of mechanism, every function under 12 lines, plus five demos. Then: 

python3 lora_from_scratch.py
import numpy as np


def lora_init(d_in, d_out, r, seed=0):
    rng = np.random.default_rng(seed)
    A = rng.normal(0, 1 / np.sqrt(d_in), (d_in, r))   # random, like any layer
    B = np.zeros((r, d_out))                           # zero: the patch is empty
    return A, B


def lora_forward(x, W, A, B, alpha, r):
    # (1) BASE: the frozen pretrained layer, untouched
    base = x @ W

    # (2) PATCH: squeeze x down to r numbers, expand back out. Two thin matmuls.
    patch = (x @ A) @ B

    # (3) SCALE by alpha/r, so changing r doesn't change how big a step is.
    #     Section 4.1: "we simply set alpha to the first r we try".
    return base + (alpha / r) * patch


def merge(W, A, B, alpha, r):
    """Fold the patch into the weight. One matrix again: zero extra latency."""
    return W + (alpha / r) * (A @ B)


def fit_delta(X, Y, W):
    """What full fine-tuning would learn: the dW that best maps X to Y."""
    residual = Y - X @ W                          # what the base gets wrong
    dW, *_ = np.linalg.lstsq(X, residual, rcond=None)
    return dW


def low_rank(dW, r):
    """Keep the top-r directions of dW. Returns A (d_in, r), B (r, d_out)."""
    U, s, Vt = np.linalg.svd(dW, full_matrices=False)
    A = U[:, :r] * s[:r]      # fold the singular values into A
    B = Vt[:r]
    return A, B, s            # A @ B is the best rank-r approximation of dW


def lora_grads(x, A, B, G, alpha, r):
    s = alpha / r
    grad_A = s * x.T @ G @ B.T
    grad_B = s * (x @ A).T @ G
    return grad_A, grad_B

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living — reverse-engineering from broken behaviour is your native mode. Every row is a one-line edit, or something the script already prints.

Break thisWatch for
In lora_init, replace np.zeros((r, d_out)) with a random matrixDemo 4: identical to base? False, outputs moved 31.5% before a single training step. Your fine-tune starts by breaking the model.
In demo 4's init table, zero A as well: np.zeros_like(A) for both|dL/dA| = 0.000, |dL/dB| = 0.000. Nothing ever moves. B = 0 only works because A is random.
In lora_forward and merge, replace (alpha / r) with 1Demo 5: the same Adam step drops the loss 78× more at r = 64 than at r = 1. Every change of r is now a change of learning rate.
In demo 2, hand make_task the full-rank dW_full instead of dW_trueSingular values 0.93, 0.88, 0.85… no cliff. Rank 8 leaves the test error at 0.38 against 0.05 for the full fine-tune. No small r saves a full-rank task.
In demo 3, delete the np.maximum(…, 0) from the adapterThe adapter residual falls from 5.44e-01 to float precision. Remove the relu and an adapter is a LoRA — it folds into W.
In demo 1, read the "every matrix" block: attention, MLP, embeddings, LM head37.7M becomes 170.9M trainable, 4.5×. The paper adapted Wq and Wv only and froze the MLP; under a fixed budget, section 7 found Wq + Wv at r = 4 beat Wq alone at r = 8.
Then read the paper — but only section 4 (equation 3 and the two paragraphs after it are this page) and section 7 (what the update looks like on GPT-3). Skip the GLUE and E2E tables in section 5 except the GPT-3 one, skip section 6. You'll find the paper is just the comments you've already read.

On the maths anxiety

Matrix shapes, one outer product, and the word rank. Rank is the number of independent directions in a matrix; the SVD is the library call that measures it. Least squares is np.linalg.lstsq, another library call. That is the whole paper. The one gradient I wrote by hand is the chain rule for a linear layer — two matmuls — and you never need to write it again; loss.backward() does it, the same way you don't hand-roll TCP.

What this page did give you is a habit: look at a weight matrix and ask how many of its directions matter. That question comes back in quantisation, pruning, KV-cache compression and every "why is this model smaller" interview answer. Not research maths.

Where to go next

This page gave you one thing: the diff is thin, in code you can break. I skipped the training loop and the hyperparameter sweeps. Two other people cover those properly.

Order I'd take them in: paper section 4 → from scratch → practical tips. If you only read one, the from-scratch one — it's this page with the training loop put back.