Quantisation: what fits on your GPU

Dettmers et al., 2022 — with Frantar et al. 2022, Lin et al. 2023, Dettmers et al. 2023

Four papers, one question: which numbers are you allowed to be sloppy about?

Download quantization_from_scratch.py — everything below runs from it. NumPy only.

I'm an AI Engineer. I've passed load_in_4bit=True to from_pretrained, watched a 13B model appear on a card that should not have held it, and had no idea what I'd just agreed to. What is NF4? Why is group_size=128 in every config file? Why does GPTQ need a calibration dataset when bitsandbytes doesn't? Nobody told me the whole field is one operation and one argument about it.

A float is a storage budget, and most of it is wasted

You already know this idea. It's JPEG. Almost none of a photograph's exactness survives being saved at quality 80, and it looks the same. You can throw away a lot before anyone notices — but throw away the wrong thing and it's obvious instantly: ringing round the text, banding across the sky.

A weight matrix is the same bargain. fp16 spends 16 bits per number on a range up to 65504, and your weights all sit between about −3 and 3. So compress them: 

scale = abs(x).max() / 127        # map your range onto 255 slots
q     = round(x / scale)          # <- the lossy step. Information dies here
store(q)                          # one byte per weight instead of two
x2    = q * scale                 # multiply back. Close to x. Not x.

That's it. That is every paper on this page. Everything else — LLM.int8(), GPTQ, AWQ, NF4 — is an argument about which numbers share a scale, where the levels sit, and which numbers you refuse to compress at all.

TermMeansProgrammer's analogy
ScaleThe float that maps your range onto the integer gridThe JPEG quality slider
Zero pointAn integer offset, so the grid need not be centred on 0An unsigned type with a bias
GranularityHow many numbers share one scale: tensor, channel, or group of 128Chunk size. Smaller chunks, more headers
OutlierOne huge value that drags the scale coarse for everyone elseOne 4 GB row in a table you're paging
PTQ / QATQuantise a finished model, vs train it knowing it will be quantisedPost-processing vs building for the target

The number that decides everything

Before any of the cleverness: parameters × bytes-per-parameter. That is the decision.

Don't take my word for it. Here's the actual output:

 model        fp16        int8        int4
    7B      13.0 GB       6.5 GB       3.3 GB
   13B      24.2 GB      12.1 GB       6.1 GB
   70B     130.4 GB      65.2 GB      32.6 GB

fits on the card? weights only, nothing left for activations:

 model    fp16 24/80    int8 24/80    int4 24/80
    7B      yes  yes      yes  yes      yes  yes
   13B       NO  yes      yes  yes      yes  yes
   70B       NO   NO       NO  yes       NO  yes

Weights are only half the budget. The KV cache is a second one, and it grows with context rather than with parameters — the arithmetic for it is on the KV cache page. Adding it at 4096 tokens, one sequence, fp16, using Llama-2 shapes: 

    7B fp16: weights   13.0 GB + KV  2.00 GB =   15.0 GB   24GB card: fits
    7B int4: weights    3.3 GB + KV  2.00 GB =    5.3 GB   24GB card: fits
   13B fp16: weights   24.2 GB + KV  3.12 GB =   27.3 GB   24GB card: no
   13B int4: weights    6.1 GB + KV  3.12 GB =    9.2 GB   24GB card: fits
   70B fp16: weights  130.4 GB + KV  1.25 GB =  131.6 GB   24GB card: no
   70B int4: weights   32.6 GB + KV  1.25 GB =   33.8 GB   24GB card: no
The 70B has the smallest KV cache of the three. That is not a typo — it uses grouped-query attention with 8 KV heads instead of 80, so its cache is smaller than the 13B's. Model size and cache size are separate budgets set by separate architecture choices.

The one formula

q = clamp(round(x / s), −2b−1+1, 2b−1−1)   x̂ = q · s   where s = max|x| / (2b−1−1)

Read it as pipeline stages:

StepShapeWhat it does
max|x|scalar (or per column, or per group)The only choice you make. Everything on this page is an argument about this line
x / s(n, d)Rescale into integer units. Now the numbers mean "how many slots"
round(n, d)The lossy step. Worst case half a slot of error, per weight
clamp(n, d)Anything past the end saturates. Cheap in int8, brutal if you clipped an outlier
q · s(n, d)Back to float, in the kernel, just before the matmul. This is weight-only quantisation

Symmetric assumes the data straddles zero. If it doesn't — a ReLU output, say — you waste the negative half of the grid, and an integer zero_point shifts the grid to fix it: 

now shift the data to [0.17, 2.60] -- nothing is negative:
  symmetric  relative error: 0.003939  (half the grid unused)
  asymmetric relative error: 0.001680  zero_point=-18
  symmetric  int8 codes: 8 .. 127 (of -127..127, 47% of the grid used)
  asymmetric int8 codes: 0 .. 255 (of 0..255, 100% used)

Where the scale lives is the whole game

One scale for a whole tensor means the loudest channel decides the resolution for every quiet one. This is the single most useful thing on this page, and it is why naive quantisation fails.

Here's a 64×8 matrix where one column has been made 50× louder and one 50× quieter, quantised to int8 both ways: 

channel absmax: [135.558  31.      2.466   1.988   2.553   2.176   2.138   0.057]

per-tensor  one scale  = 1.067387
per-channel scales     = [1.067 0.244 0.019 0.016 0.02  0.017 0.017 0.   ]

per-channel error, column by column:
  col 0: per-tensor  0.007493   per-channel 0.007493
  col 2: per-tensor  0.290450   per-channel 0.005758
  col 7: per-tensor  1.000000   per-channel 0.007100

distinct int8 codes in the quietest column (col 7):
  per-tensor : 1 of 255
  per-channel: 53 of 255

Column 7 has relative error 1.0. Every value in it rounded to zero. The column is gone. It cost one float per column to save it — eight floats for the whole matrix.

Group-wise scales are the same idea at finer grain: chop each column into groups of g weights and give each group a scale. Here is 4096 weights at int4, eight of them 30× larger than the rest: 

   group   rel error   scales   bits/weight
   whole    0.662549        1         4.004
     128    0.257297       32         4.125
      64    0.187432       64         4.250
      32    0.146733      128         4.500

same sweep with NO outliers, so you can see what the outliers cost:
   whole    0.152834
     128    0.116649
Now you know why every config file says group_size: 128. It buys most of the accuracy back for 0.125 extra bits per weight — one fp16 scale per 128 weights. Going to 32 costs four times the overhead for a fraction more. QLoRA's double quantisation then quantises those scales too, which the paper reports saves about 0.37 bits per parameter.

LLM.int8(): the outliers are systematic

Dettmers et al. found something that is not obvious: past roughly 6.7B parameters, transformers develop outlier features — a small number of hidden dimensions whose activations are far larger than everything else, and they appear in the same dimensions across layers and across inputs. They are not noise. Zero them out and the model's performance collapses.

They're also fatal for per-tensor int8, because the scale is set by the max. One huge value spends the whole grid on empty space. Here are 32 tokens × 16 dimensions, with outliers injected into dims 3 and 11: 

clean activations, absmax 3.056
  per-tensor int8 relative error: 0.006828

with outliers in dims 3 and 11, absmax 105.047
  per-tensor scale went 0.024066 -> 0.827141  (34.4x coarser)
  per-tensor int8 relative error: 0.017330
  error on the NON-outlier dims only: 0.243144
  distinct int8 codes used by dim 0: 5 of 255

mixed precision: keep columns [3, 11] in 16-bit,
  that is 2/16 = 12.5% of the dimensions
  relative error, whole tensor : 0.000466
  relative error, non-outlier  : 0.006992

Whole-tensor error looks like 1.7%, which is survivable. That average is a lie: it's dominated by the two outliers, which are huge and quantise fine. The dimensions that matter to everything else are at 24% error, and dimension 0 is using five of its 255 codes. The damage is hidden by the metric.

LLM.int8()'s fix is not clever, and that's the point. Pull the outlier columns out, do those in fp16, do the other 99.9% in int8, add the two results. One mixed-precision matmul. The paper's claim is that this holds full 16-bit quality while halving the weight memory — the first int8 method that survived 175B scale.

This is where weight quantisation and activation quantisation part company. Weights are well-behaved: bounded, roughly normal, known ahead of time. Activations depend on the input and are where the outliers live. That is why GPTQ, AWQ and NF4 are all weight-only — they shrink the checkpoint and dequantise before the matmul. LLM.int8() is doing the harder thing.

GPTQ: quantise in an order, and apologise as you go

Everything above quantises every weight independently, in one shot, and accepts the error. GPTQ asks a better question: when I round this weight down, can I nudge the weights I haven't done yet to compensate?

It goes column by column. Quantise a column, measure the error, then update the columns not yet done so they absorb it — using approximate second-order information (a Hessian, estimated from a small calibration set) to decide how. Each rounding decision leaves a debt, and the not-yet-frozen weights pay it.

Three things follow, and they are the interview material. It is one-shot post-training: no retraining, no gradients, no labels. It needs calibration data, which bitsandbytes' int8 and NF4 don't — the usual reason a GPTQ build is awkward in a pipeline. And it is weight-only, 3 to 4 bit: the paper's headline is 3 and 4-bit quantisation up to 175B with negligible accuracy loss, which is why GPTQ checkpoints are everywhere on the Hub. The Hessian update is not in my script — it needs a calibration set and a Cholesky, and it would bury the idea.

AWQ: protect the weights the activations care about

AWQ's finding: a small fraction of weights are salient, and protecting roughly 1% of them recovers most of the quality. The non-obvious part is how you find them. Not by weight magnitude — by activation magnitude. The weights that matter are the ones multiplying the input channels carrying large activations. That is the LLM.int8() finding again, seen from the weight side.

And AWQ does not keep them in fp16, because mixed precision is ugly in a kernel: ragged layout, slow paths. Instead it scales the salient channels up before quantising and scales the matching activations down, so the maths is unchanged and the important weights land on a finer part of the grid. Everything stays uniform int4, which is why AWQ kernels are fast.

The distinction to keep straight: LLM.int8() is mixed-precision and finds outliers at runtime, per batch, in the activations. AWQ is uniform int4 and uses activation statistics once, at quantisation time, to decide how to scale the weights. GPTQ uses a calibration set to correct rounding error. Three answers to "which numbers do I protect", and no backprop in any of them.

QLoRA and NF4: put the levels where the data is

Every scheme so far spaces its levels evenly, and QLoRA's argument is that this is obviously wrong. Pretrained weights are roughly normally distributed, piled up around zero, so evenly spaced levels spend most of their resolution in the tails where almost nothing lives. 4-bit NormalFloat puts the 16 levels at equal-probability quantiles of a normal distribution instead. Same 4 bits, levels crowded where the weights are — here are both sets, and the error on 100,000 draws from N(0,1): 

linear int4 levels (evenly spaced):
  [-1.    -0.867 -0.733 -0.6   -0.467 -0.333 -0.2   -0.067  0.067 ...]
normal-quantile levels (NF4-style):
  [-1.    -0.708 -0.542 -0.417 -0.311 -0.216 -0.127 -0.042  0.042 ...]

weights ~ N(0,1), n=100000, absmax 4.369
  linear int4 relative error: 0.167915
  NF4-style   relative error: 0.118978
  improvement: 29.1%

how much data each of the 16 levels actually gets:
  linear: emptiest level      1  busiest  22082  levels under 1% of the data: 8/16
  NF4   : emptiest level     10  busiest  14431  levels under 1% of the data: 4/16

Half the linear levels are holding under 1% of the data each. One of them caught a single weight out of a hundred thousand. That is half your bit budget spent on the tails.

Honest caveat: the real NF4 table in the paper is built asymmetrically, so that exact zero is one of the levels and zero-padding stays exact. Mine is the same construction idea in five lines, and the error numbers above are from my version, not theirs.

Why QLoRA needs LoRA

Here is the bit that made QLoRA click for me. Once weights are on a 4-bit grid you cannot fine-tune them. A gradient step smaller than half the grid step rounds straight back to where it started: 

4-bit grid step (the scale) = 0.189194

after one SGD step at lr=0.001:
  moved by         : [-0.001 -0.002 -0.     0.001  0.001 -0.002 -0.     0.002]
  after re-quantise: [0. 0. 0. 0. 0. 0. 0. 0.]
  weights that actually changed: 0 of 8

same gradient at lr=0.5: 7 of 8 weights changed
  step sizes taken : [-0.378  0.    -0.189  0.568  0.189 -0.757 -0.189  0.946]

frozen 4-bit base + fp16 LoRA adapter, same lr=0.001:
  adapter output after : max |delta| = 9.424e-07
  weights that actually changed: 8 of 8, by steps smaller than
  the 4-bit grid step of 0.189194

The update silently disappears. Turn the learning rate up until something moves and every step is a whole grid step — that isn't fine-tuning, it's vandalism. So QLoRA freezes the 4-bit base and trains a small higher-precision LoRA adapter on top of it. Gradients flow through the frozen quantised weights into the adapter; the base is never updated. I won't re-explain LoRA — that page does.

The paper's three contributions: NF4; double quantisation, which quantises the block scales themselves; and paged optimisers, which use unified memory to survive the gradient-checkpointing spikes that would otherwise OOM you. The headline result is finetuning a 65B model on one 48GB GPU without degrading 16-bit finetuning performance.

The code

Pure NumPy. No torch, no autograd, no training — quantisation is arithmetic, and you can watch every rounded number. Every function under 12 lines.

Download the full script — about 70 lines of core plus seven runnable demos. (The asymmetric pair, quantize_asymmetric and its inverse, is in the file; it is the same four lines with a zero_point.) Then: 

python3 quantization_from_scratch.py
def quantize_symmetric(x, bits=8):
    qmax = 2 ** (bits - 1) - 1              # 127 for int8
    scale = np.abs(x).max() / qmax          # one float per tensor
    q = np.clip(np.round(x / scale), -qmax, qmax)   # the lossy step
    return q.astype(np.int32), scale

def dequantize_symmetric(q, scale):
    return q * scale                        # back to float. Not the original.

def quantize_per_channel(W, bits=8, axis=0):
    qmax = 2 ** (bits - 1) - 1
    scale = np.abs(W).max(axis=axis, keepdims=True) / qmax   # one per column
    q = np.clip(np.round(W / scale), -qmax, qmax)
    return q.astype(np.int32), scale

def quantize_grouped(x, bits=4, group=128):
    flat = x.reshape(-1, group)                     # rows of g values
    qmax = 2 ** (bits - 1) - 1
    scale = np.abs(flat).max(axis=1, keepdims=True) / qmax
    q = np.clip(np.round(flat / scale), -qmax, qmax)
    return (q * scale).reshape(x.shape), scale.size  # recovered, n_scales

def normal_levels(bits=4):
    n = 2 ** bits
    ps = [(i + 0.5) / n for i in range(n)]          # equal probability mass
    lv = np.array([NormalDist().inv_cdf(p) for p in ps])
    return lv / np.abs(lv).max()                    # squash into [-1, 1]

def quantize_to_levels(x, levels):
    """Snap each value to the nearest level, after absmax-scaling to [-1, 1]."""
    scale = np.abs(x).max()
    idx = np.abs(x[:, None] / scale - levels[None, :]).argmin(axis=1)
    return levels[idx] * scale

def outlier_columns(X, threshold=6.0):
    return np.where(np.abs(X).max(axis=0) >= threshold)[0]

def quantize_mixed(X, outlier_cols, bits=8):
    keep = np.zeros(X.shape[1], dtype=bool)
    keep[outlier_cols] = True
    out = X.copy()                                  # outlier cols stay fp16-ish
    rest = X[:, ~keep]
    q, s = quantize_symmetric(rest, bits)           # the rest goes to int8
    out[:, ~keep] = dequantize_symmetric(q, s)
    return out

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living — and quantisation is unusually good for this, because every failure shows up as a number you can print rather than a training curve you wait three hours for.

Break thisWatch for
In demo_2, quantise with quantize_symmetric instead of quantize_per_channelColumn 7 goes to relative error 1.000000 and uses 1 of 255 int8 codes. The whole channel rounds to zero.
In demo_1, run the shifted data y = x + 1.0 through quantize_symmetric onlyError 0.003939 vs 0.001680, and only 47% of the int8 grid is used. Half the range is spent on negatives that never occur.
In demo_6, use only group=n (one scale for everything) at 4 bitsRelative error 0.662549 against 0.257297 at group=128. Eight big weights ruin four thousand small ones.
In demo_3, change outlier_columns(X, threshold=6.0) to threshold=1000.0No columns are kept in 16-bit, and non-outlier error jumps from 0.006992 back to 0.243144. That one line is LLM.int8().
In demo_5, pass linear_levels(4) where normal_levels(4) is usedError 0.118978 → 0.167915, and the levels holding under 1% of the data go from 4/16 to 8/16. Half your levels sit where there's no data.
In demo_7, try to train the 4-bit weights directly: keep lr=1e-3 and re-quantise after the step0 of 8 weights change. Not "trains badly" — the update is silently discarded. This is exactly why QLoRA trains a LoRA adapter instead.
Then read the papers, and not all of them. LLM.int8() first, the section on emergent outlier features — that is the finding everything else reacts to. Then QLoRA's method section, for NF4 and double quantisation. GPTQ and AWQ: method sections only. Skip the perplexity tables in all four unless you are choosing between checkpoints today, in which case read them and trust nobody's summary of them, mine included — there are no perplexity numbers on this page on purpose.

On the maths anxiety

There is almost none here. Division, rounding, clamping, and one idea about where to put 16 tick marks on a number line. If you can write round(x / scale) you have implemented the core of every paper on this page. No torch, no autograd, no GPU.

Two places real maths does appear, and you can leave both to the library. GPTQ's compensation step uses second-order information — a Hessian, an approximation of how sensitive the loss is to each weight — and the implementation involves a Cholesky decomposition for numerical stability. NF4's levels come from the inverse normal CDF. I called NormalDist().inv_cdf from the standard library and moved on.

Worth your time instead: the arithmetic in the memory section, and the vocabulary — weight-only vs activation quantisation, post-training vs quantisation-aware training, per-tensor vs per-channel vs per-group, symmetric vs asymmetric. Those distinctions are what the questions are actually about, and none of them need a proof.

Where to go next

This page gave you one thing: the operation, and the four arguments about it, in code you can break. What it skipped is the part that decides your production choice — measured quality, and kernel speed.

Order I'd take them in: the bitsandbytes 8-bit post first, because it makes the outlier finding visual; then the LLM.int8() paper itself; then the 4-bit post and QLoRA together, once the LoRA page is fresh; GPTQ and AWQ last, when you're actually choosing a checkpoint.