Temperature, top-k and top-p: the three knobs on the API

Holtzman et al., 2019 · Fan et al., 2018

Three parameters everyone has typed into a request body and nobody has explained together. They are two operations, in a fixed order.

Download sampling_from_scratch.py — everything below runs from it. NumPy only, no model needed.

I shipped an LLM feature with temperature=0.7, top_p=0.9 in the config for about eight months before I could have told you what the second one did. I knew the folklore — "lower temperature is more factual", "top_p is like temperature but different" — and the folklore is wrong in a way that matters. My mental model was three dials on the same box. It isn't. It's a contrast slider and a crop tool, applied in that order, and once you see the order the inconsistent intuitions all resolve.

Temperature is contrast. Top-k and top-p are the crop.

At every step the model hands you one array: a score per vocabulary entry. Sixteen entries in my toy vocab, fifty thousand in a real one. That's it. That array is the only thing these three parameters touch. 

logits = model(prompt)            # one float per vocab entry, unbounded
probs  = softmax(logits)          # percentages, summing to 1
token  = draw_one(probs)          # weighted dice roll

Temperature goes in on line 1 — it divides the logits before the softmax. Top-k and top-p go in on line 3 — they delete entries from probs and renormalise what's left. Different stage, different kind of operation: 

logits = model(prompt) / T                     # CONTRAST: gaps stretch/squash
probs  = softmax(logits)
probs  = keep_top_k(probs, k)                  # CROP: a fixed count survives
probs  = keep_top_p(probs, p)                  # CROP: a fixed mass survives
probs  = probs / probs.sum()                   # what's left must sum to 1
token  = draw_one(probs)
KnobWhat it actually isProgrammer's analogy
temperatureDivide the logits. Ranking is preserved; only the gaps change.The contrast slider. Same photo, more or less separation between tones.
top-kKeep the k highest entries. A fixed count.LIMIT 40. Blind to how good row 41 was.
top-pKeep the smallest set whose probabilities add up to p. A fixed mass.Pagination by bytes, not by rows. The row count falls out of the data.
The decoding pipeline, in execution order Temperature divides the raw logits above the softmax boundary. Below it, on probabilities, top-k then top-p delete entries, the survivors are renormalised, and one token is drawn. logit space — unbounded floats logits ÷ temperature ranking cannotchange here probability space — after softmax, sums to 1 softmax top-k top-p renormalise after, not before draw one token Two operations, one order. Temperature reshapes. k and p delete.Swap them and you get a different set.
The dashed line is the softmax. Everything a temperature setting does happens above it; everything top_k and top_p do happens below. That single boundary is the whole confusion.

Why Holtzman's paper is the one that matters

Before 2019 the received wisdom was that better decoding meant getting closer to the most likely sequence: beam search, wider beams, greedy at the limit. The Curious Case of Neural Text Degeneration is the paper that said the goal itself is wrong. Maximising likelihood produces text that repeats and then locks — the same clause over and over — and the authors' central observation is that human-written text does not sit at the top of the model's probability distribution at all. It wanders through a lower-probability band that greedy decoding never visits.

So the fix isn't a better search. It's to sample, but from a truncated distribution — and to truncate by mass rather than by count, because the right count changes at every step. That's nucleus sampling. Top-k, from Fan et al.'s story-generation paper the year before, is the thing it improves on.

The one formula

V(p) = smallest V′ ⊆ V such that Σx∈V′ P(x) ≥ p

Read it as pipeline stages:

StepShapeWhat it does
logits / T(V,)Temperature. On the raw scores, before anything else.
softmax(V,)Percentages summing to 1. The boundary.
argsort(−p)(V,)Most likely first. Both crops need the ranking.
cumsum(V,)Running total. Where it crosses p, you stop.
keep, zero rest(V,)The nucleus. Everything outside it is unreachable.
/ sum(V,)Renormalise. After truncating, never before.

Temperature never reorders anything

Dividing every logit by the same positive number is monotonic. Whatever was the most likely token stays the most likely token. Temperature cannot promote a word into first place; it can only decide how big a lead first place has. That's why "raise the temperature to get a different answer" works only in combination with the dice roll — you widened the gap the dice have to cover, you didn't change the ranking.

Don't take my word for it. Here's the actual output:

  T=0.2  H=0.0980  Monday=0.9811  tomorrow=0.0180  next=0.0009  the=0.0001
  T=0.5  H=0.7347  Monday=0.7743  tomorrow=0.1563  next=0.0471  the=0.0173
  T=0.8  H=1.2322  Monday=0.5862  tomorrow=0.2156  next=0.1019  the=0.0545
  T=1.0  H=1.4843  Monday=0.4967  tomorrow=0.2232  next=0.1225  the=0.0743
  T=1.5  H=1.9354  Monday=0.3525  tomorrow=0.2068  next=0.1386  the=0.0993
  T=2.0  H=2.2101  Monday=0.2717  tomorrow=0.1821  next=0.1349  the=0.1051

  ranking is IDENTICAL at every temperature:
    all six orderings equal? True
    only the GAPS changed. temperature cannot promote a token.

H is entropy in nats — how undecided the distribution is. At T=0.2 it's 0.098, which is "I have already decided". At T=2.0 it's 2.21, against a ceiling of ln 16 ≈ 2.77 for a 16-token vocabulary, which is nearly "no idea". Same model, same prompt, same ranking.

The trap at T = 0

temperature=0 is in every API doc and it is not a temperature. Dividing by zero gives you infinities, then nan out of the softmax, then a crash in the sampler. Every real implementation branches around it: 

  T = 0 exactly:
    raises: T=0: real code branches to argmax instead
    raw numpy would give: [6.0e+12 5.2e+12 4.6e+12] ... then inf/nan
    what real code does instead: argmax -> 'Monday' ✓

So "temperature 0" in a request body means "skip this whole page and take the argmax". It is a special case wearing a number's clothing. The interview question here is not the division; it's whether you know T = 0 means greedy, and that greedy means the failure mode in the last section.

Top-k: a fixed count, blind to shape

Keep the k highest-probability tokens, zero the rest, renormalise. On my confident step — "the meeting is scheduled for ___", where Monday takes half the mass — small k works fine: 

  k=1   keeps 0.4967 of the mass | Monday=1.0000
  k=3   keeps 0.8424 of the mass | Monday=0.5896 tomorrow=0.2649 next=0.1454
  k=5   keeps 0.9468 of the mass | Monday=0.5246 tomorrow=0.2357 next=0.1294 ...
  k=10  keeps 0.9952 of the mass | Monday=0.4991 tomorrow=0.2242 next=0.1231 ...

Note what renormalising does: at k=3, Monday goes from 0.4967 to 0.5896. The survivors absorb the deleted mass in proportion to what they already had. Nothing is "boosted"; the denominator shrank.

Now the same k on a second step, where the model genuinely doesn't know what comes next — a flat distribution with entropy 2.7374 nats against that 2.77 ceiling: 

  k=3   confident step keeps 0.8424 | flat step keeps 0.2616  (gap 0.5807)
  k=5   confident step keeps 0.9468 | flat step keeps 0.4117  (gap 0.5351)
  k=10  confident step keeps 0.9952 | flat step keeps 0.7275  (gap 0.2678)

One k=3 setting is two completely different behaviours. On the confident step it keeps 84% of what the model believes — reasonable. On the flat step it keeps 26% and throws away three quarters of a distribution the model was telling you was genuinely open. Turn k up to fix the flat step and you have now let junk into the confident one. There is no value of k that is right for both, and a real generation makes both kinds of step in the same sentence.

Top-p: a fixed mass, so the set size adapts

Nucleus sampling asks a different question. Not "how many tokens?" but "how much of the model's belief?". Sort, take a running total, stop when it crosses p. The size of the surviving set is decided by the distribution, not by you. Same two steps, same script: 

  p     confident step        flat step
  0.5    2 tokens, mass 0.7199    7 tokens, mass 0.5476
  0.8    3 tokens, mass 0.8424   12 tokens, mass 0.8333
  0.9    4 tokens, mass 0.9166   14 tokens, mass 0.9253
  0.95   6 tokens, mass 0.9671   15 tokens, mass 0.9654
  1.0   16 tokens, mass 1.0000   16 tokens, mass 1.0000

That p=0.9 row is the paper. One setting, four tokens where the model is confident and fourteen where it isn't. The nucleus grew because the distribution flattened — nobody adjusted anything. 

  at p=0.9 the nucleus is 4 tokens on the confident step and 14 on the flat one.
    confident nucleus: Monday tomorrow next the
    flat nucleus:      Monday tomorrow next the March later noon sometime ...
One distribution, three treatments Sixteen bars for the same confident step, drawn three times: untouched, cropped at a fixed count of three, and cropped at a cumulative mass of 0.9 which keeps four. Filled bars survive; pale bars are deleted. The labels give the mass each crop keeps on this step and on a flatter one. untouched all 16 reachable tail mass 0.0111 top-k, k=3 cut keeps 0.8424 here but 0.2616 when flat top-p, p=0.9 cut keeps 4 tokens here but 14 when flat filled = survives the crop, pale = deleted then renormalised away
Same sixteen bars every time — this is the confident step, straight from the script. k=3 cuts at a place it chose before seeing the data. p=0.9 cuts one bar later here, and ten bars later on the flat step. Both crops then renormalise, so all three panels end up summing to 1.
p = 1.0 is not "off", it is "keep everything". On my 16-token vocab that leaves 0.0111 of mass in tokens like asdf at 0.000167 — reachable roughly once every six thousand tokens. On a 50k vocab that tail is where the genuinely bizarre outputs come from. Holtzman's argument is precisely that this tail is disproportionately reachable when you sample untruncated.

The order is part of the definition

Because temperature reshapes and truncation crops, running them in the other order gives you a different distribution — not a rounding difference, a different set of reachable tokens. At T=1.5 and p=0.9 on my fixed vector: 

  T=1.5, p=0.9
    temp-then-p keeps 7 tokens: Monday tomorrow next the March later noon
    p-then-temp keeps 4 tokens: Monday tomorrow next the
    same surviving set? False
       Monday  temp-then-p 0.3809   p-then-temp 0.4422   diff -0.0613
     tomorrow  temp-then-p 0.2234   p-then-temp 0.2594   diff -0.0359
         next  temp-then-p 0.1498   p-then-temp 0.1739   diff -0.0241
          the  temp-then-p 0.1073   p-then-temp 0.1246   diff -0.0173
        March  temp-then-p 0.0589   p-then-temp 0.0000   diff +0.0589
  max abs difference: 0.0613

March is reachable in one order and impossible in the other. Real implementations — Hugging Face's logits processors, vLLM, the hosted APIs — do temperature, then top-k, then top-p. Raising temperature flattens the distribution first, which makes the nucleus grow, which is what you expected when you turned the dial up. In the wrong order the nucleus is frozen at T=1 and temperature only redistributes inside it. Same two numbers, quietly different product.

The other order mistake is renormalising before you truncate instead of after. It doesn't error — it just hands the sampler a vector that no longer sums to 1: 

  and renormalising BEFORE truncating instead of after:
    sum after cutting an already-normalised vector: 0.9807
    np.random.choice raises: probabilities do not sum to 1

Why greedy decoding actually fails

You don't need a language model to see degeneration. A first-order Markov chain over eight words is enough: take the argmax at every step and the moment you revisit a state you are in a cycle forever, because the argmax is deterministic. Nothing can break it. This is the same mechanism as the repetition loop in a real model, minus the excuses. 

  greedy from 'the': the model is a model is a model is a model is ...
  loop detected, period 3: model is a
  greedy avg surprisal: 1.0670 bits/token

  sampled (T=1.0, p=0.9): the model is the the data good model learns from data ...
  loop detected? no -- it revisits states without getting stuck
  sampled avg surprisal over 200 runs: 2.0463 bits/token
  greedy is 0.9792 bits/token LESS surprising
  -- and unreadable. maximising likelihood is not the goal.

Read those two surprisal numbers again. Surprisal is −log₂ P(token) under the model — the number the model is trained to make small. Greedy wins on it, by a full bit per token, and produces "model is a model is a model is a". The sampled text is twice as surprising to the model and is the one you'd ship.

That gap is Holtzman's finding, in miniature. Human text is not the most likely text. A decoder that optimises for likelihood optimises for the wrong thing, and the failure is not subtle — it's a literal loop. The paper's response is to keep the sampling and constrain where you sample from.

And the parameter that reproduces this on purpose: top_k=1. Keeping exactly one token means the renormalised distribution is 1.0 on the argmax, so the dice roll is a formality. 

  top-k with k=1 is greedy wearing a hat: identical to the greedy walk
    k=1 walk: the model is a model is a model is a model is ...
None of this changes the model. Same weights, same logits, same forward pass. Temperature, k and p are how you read the output. That's why they're request parameters and not a fine-tune, and why "the model is repeating itself" is often a decoding bug, not a training one.

The code

Pure NumPy. No torch, no model, no training — sampling is arithmetic over a probability vector, so the whole mechanism is six short functions and you can watch every number move. Every function under 20 lines.

Download the full script — 309 lines plus 6 runnable demos. Then: 

python3 sampling_from_scratch.py
# ---------------------------------------------------------------------------
# STAGE 0 -- softmax. Turns a row of logits into percentages summing to 1.
#
# The `- max` is not maths, it's overflow protection: exp(1000) is inf,
# exp(1000-1000) is 1, and the ratio is unchanged.
# ---------------------------------------------------------------------------
def softmax(logits):
    z = logits - np.max(logits)
    e = np.exp(z)
    return e / e.sum()


# ---------------------------------------------------------------------------
# STAGE 1 -- TEMPERATURE. It divides the LOGITS, before the softmax.
#
# T < 1 stretches the gaps apart  -> sharper, closer to argmax.
# T > 1 squashes the gaps together -> flatter, closer to uniform.
# It is monotonic, so it can never reorder the vocabulary: the most likely
# token before is the most likely token after. It is a contrast slider.
#
# T = 0 is division by zero. Every real implementation special-cases it to
# mean "greedy" instead of computing it.
# ---------------------------------------------------------------------------
def apply_temperature(logits, T):
    if T == 0:
        raise ZeroDivisionError("T=0: real code branches to argmax instead")
    return logits / T


# ---------------------------------------------------------------------------
# STAGE 2 -- TOP-K (Fan et al., 2018). Keep the k highest-probability tokens,
# throw the rest away, renormalise what is left back to sum 1.
#
# k is a fixed COUNT. It has no idea what shape the distribution is, so the
# probability mass it happens to capture swings wildly step to step.
# ---------------------------------------------------------------------------
def top_k(p, k):
    keep = np.argsort(-p)[:k]              # indices of the k largest
    out = np.zeros_like(p)
    out[keep] = p[keep]
    return out / out.sum(), np.sort(keep)  # RENORMALISE AFTER truncating


# ---------------------------------------------------------------------------
# STAGE 3 -- TOP-P / nucleus sampling (Holtzman et al., 2019). Keep the
# SMALLEST set whose cumulative probability reaches p, then renormalise.
#
# p is a fixed MASS, so the size of the set is decided by the distribution,
# not by you. Confident step -> a couple of tokens. Flat step -> dozens.
# That adaptivity is the whole paper.
# ---------------------------------------------------------------------------
def top_p(p, p_thresh):
    order = np.argsort(-p)                 # most likely first
    cum = np.cumsum(p[order])
    n = int(np.searchsorted(cum, p_thresh) + 1)  # include the crossing token
    n = min(n, len(p))
    keep = order[:n]
    out = np.zeros_like(p)
    out[keep] = p[keep]
    return out / out.sum(), np.sort(keep)


# ---------------------------------------------------------------------------
# STAGE 4 -- the pipeline every inference server runs, in this order:
#     logits -> temperature -> softmax -> top-k -> top-p -> renormalise -> draw
# Temperature acts on logits. Truncation acts on probabilities. Swap those
# two and you get a different distribution, not a rounding difference.
# ---------------------------------------------------------------------------
def sample_once(logits, T=1.0, k=None, p_thresh=None, rng=None):
    rng = rng or np.random.default_rng(0)
    probs = softmax(logits) if T == 0 else softmax(apply_temperature(logits, T))
    if T == 0:
        return int(np.argmax(logits))      # the special case, not a division
    if k is not None:
        probs, _ = top_k(probs, k)
    if p_thresh is not None:
        probs, _ = top_p(probs, p_thresh)
    return int(rng.choice(len(probs), p=probs))

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living — reverse-engineering from broken behaviour is your native mode, and every one of these is a bug I have watched reach production.

Break thisWatch for
Delete the if T == 0 guard in apply_temperature and call it with 0.0logits/0 gives inf, softmax returns nan, rng.choice raises. What you wanted was np.argmax → Monday.
Call walk(0, 60, "sample", k=1)Byte-identical to the greedy walk: the model is a model is a…, loop of period 3. k=1 is greedy with extra steps.
In demo_5, truncate first and apply temperature to the survivorsThe nucleus shrinks from 7 tokens to 4. March goes from 0.0589 to 0.0000 — reachable in one order, impossible in the other.
Set p_thresh=1.0 in top_pNothing is cut. 0.0111 of mass stays in the junk tail; asdf at 0.000167 is still drawable.
In top_k, swap the last line to out / p.sum() — normalise by the pre-cut totalThe returned vector sums to 0.9468 at k=5, not 1.0. NumPy raises "probabilities do not sum to 1"; a hand-rolled sampler would just under-sample the last token silently.
In top_p, drop the + 1 in the searchsorted lineThe token that crosses p is excluded, so p=0.5 on the confident step keeps 1 token, not 2 — silently greedy at every confident step.
Then read Holtzman's paper — the method section where the nucleus is defined, and the analysis of why beam search degenerates. The human-versus-machine probability plot early in the paper is the single image worth the download. Skip the crowdsourced evaluation tables unless you care about how text quality was scored in 2019. You'll find the paper is just the comments you've already read.

On the maths anxiety

There is almost none here, and that is worth saying plainly, because "nucleus sampling" sounds like it needs measure theory. It needs: division, a sort, a running total, and a comparison. If you can write sorted(rows, reverse=True) and stop a loop when a counter crosses a threshold, you have implemented the paper.

Two symbols worth owning. Entropy is the average surprisal of a distribution — one number for "how undecided is the model", in nats if you use ln and bits if you use log₂. Its ceiling is ln V for a V-token vocabulary, so on my 16 tokens 2.77 means uniform and 0 means certain. Surprisal is the same idea for a single token, −log P(token): the log turns a product of probabilities into a sum, which is the only reason logs are in here at all.

What you cannot skip is the shape discipline: which array is logits and which is probabilities. Almost every decoding bug I've written was applying an operation on the wrong side of the softmax. That's not maths, that's types.

Where to go next

This page gave you one thing: the three parameters as two operations in a fixed order, in code you can break. It skipped beam search entirely, skipped every benchmark in the paper, and skipped the newer samplers. Here's where those live.

Order I'd take them in: Hugging Face first, because it turns these numbers back into sentences, then Holtzman for why, then Labonne if beam search still bothers you. Fan et al. is optional context. On this site, GPT is where the temperature sweep came from and shows the loop in a real generation loop; constrained decoding is the sibling — it masks the same logits vector, but for grammar rather than for quality; and attention is where the logits are produced in the first place.