Papers
No proofs. No summations you have to squint at. Just the idea, translated into something you could set a breakpoint in.
How to read these: don't. Not first, anyway. Run the code, break it the way each page tells you to, then come back and read. Watching something fail teaches faster than watching someone explain.
ai-ml/transformers/
-
Byte-Pair Encoding: the tokeniser you inherited
Train BPE in a for loop, then watch why the model can't count letters and why the same sentence costs several times more in Tamil.
-
Attention Is All You Need
Attention is a dictionary lookup with fuzzy keys — everything else is plumbing. Runnable NumPy, four demos that prove it by breaking it.
-
RoFormer: Rotary Position Embedding
Rotate q and k instead of adding a position vector, and the dot product only sees distance. Every claim is a printed number.
-
BERT: Deep Bidirectional Transformers
Remove the causal mask, hide 15% of the tokens, guess them. Then bolt a head on per task. A tiny masked model, trained live.
-
GPT: Generative Pre-trained Transformers
Decoder-only, next token, then scale. The generation loop is a while loop. A character-level GPT trained in your terminal.
-
An Image Is Worth 16×16 Words (ViT)
The whole paper is a reshape: cut the image into patches and it's a sentence. Shapes printed at every step.
-
LLaMA: Open and Efficient Foundation Models
Not a new idea, a recipe: four deltas on the 2017 transformer and far more tokens than the scaling laws said. Parameter arithmetic included.
ai-ml/scaling/
-
Chinchilla: how big a model should you train?
Fixed compute budget: spend it on a bigger model or more data? Roughly twenty tokens per parameter, swept and plotted.
-
Mixture of Experts: the Sparsely-Gated Layer
A load balancer in front of eight small networks. Eight times the parameters, twice the compute. Hot shards and dropped tokens, as in production.
-
DeepSeek-V3: the recipe, and the interconnect that shaped it
What you build when your GPUs have the compute but not the bandwidth. Every choice traces to one halved interconnect.
ai-ml/inference/
-
The GPU, for people who only ever call .cuda()
One number, arithmetic intensity, predicts which of your operations will be slow before you profile anything.
-
KV caching and the generation loop
Why inference is affordable. The causal mask makes keys and values immutable, so you memoise them. Includes the cache-size arithmetic that decides what you can serve.
-
Grouped-Query Attention: one set of keys for several heads
The default in every modern open model. One repeat operation, and getting it wrong produces plausible garbage with no error.
-
Multi-head Latent Attention: the KV cache, compressed
Cache one small latent instead of every key and value, and fold the decompression into the query projection so it costs nothing.
-
FlashAttention: attention was never compute-bound
The n² cost is memory traffic, not arithmetic. Tile it, keep a running softmax, never store the score matrix.
-
Quantisation: what fits on your GPU
Divide, round, store. LLM.int8, GPTQ, AWQ and QLoRA's NF4 are four arguments about which numbers you may be sloppy with.
-
Speculative decoding: guess, then check
A cheap model guesses ahead, the big one verifies in a single pass. Branch prediction for tokens, with the output provably unchanged.
-
How a serving engine actually works
Continuous batching and paged memory. Your inference server is an operating system, and this is its scheduler and allocator.
ai-ml/fine-tuning/
-
LoRA: Low-Rank Adaptation
A fine-tune is a diff on the weights, and the diff is low-rank. Store the patch, not the repo. Merge it for free at inference.
-
Parameter-Efficient Fine-Tuning: the family
One question five ways: where do you insert the small trainable thing? Adapters, prefixes, prompts, (IA)³ and LoRA, with parameter counts.
ai-ml/alignment/
-
Reinforcement Learning from Human Feedback
You can't write the loss for "helpful", but you can ask which of two answers is better. Reward hacking, reproduced on purpose.
-
InstructGPT: Following Instructions with Human Feedback
The three-stage recipe that turned a completion engine into an assistant. Each stage's gain in one table. This is ChatGPT's recipe.
-
Direct Preference Optimization
The RLHF objective has an exact answer, and rearranging it makes the reward model disappear. Preference tuning becomes one classifier loss.
-
GRPO: reinforcement learning without a critic
PPO needs a second model to guess the baseline. Sample a group of answers and use their own mean instead. The algorithm behind R1.
-
DeepSeek-R1: reasoning nobody taught it
Reward only correct answers and long reasoning appears on its own, because thinking longer was the cheapest way to be right.
ai-ml/generative/
-
Auto-Encoding Variational Bayes (VAE)
An autoencoder whose bottleneck is a distribution. The reparameterisation trick is moving the random number outside the function.
-
Generative Adversarial Nets
A forger and a detective train each other. Mode collapse and vanishing gradients, triggered deliberately on a toy problem.
-
Denoising Diffusion Probabilistic Models
Learn to undo a noise-adding for-loop one step at a time. The forward process is a formula; the model is a one-step undo.
ai-ml/retrieval/
-
Sentence-BERT and CLIP: making distance mean something
Where embeddings come from. A model trained for something else gives a useless distance, measured and then fixed with contrastive loss.
-
Retrieval-Augmented Generation
An open-book exam. The retriever is the index, the prompt is the page you slide across. The failures are retrieval bugs, not model bugs.
ai-ml/prompting/
-
Chain-of-Thought and ReAct
A model gets fixed compute per token. Thinking out loud is how it buys more, and ReAct is how it buys facts.
-
Tool calling, MCP and RAG: three answers to different questions
Compared as rivals, but they solve different problems. Retrieve a fact, take an action, or standardise the wiring.
ai-ml/production/
-
Temperature, top-k and top-p: the three knobs on the API
Not three settings for one thing. Temperature reshapes the whole distribution, then k and p crop it, and the order matters.
-
Constrained decoding: making the model return valid JSON
Stop retrying until it parses. Mask the tokens that cannot continue a valid document, and invalid output becomes unrepresentable.
-
Hallucination and grounding: the model was never trying to be right
Nothing in training rewarded being correct. Why a fluent wrong answer is the objective working, and what grounding actually buys.
-
Evaluation: the part everyone skips
How big a golden set has to be before a difference means anything, and what a biased judge does to your numbers.
-
Prompt injection: why the model cannot tell your instructions from theirs
Instructions and data share one channel, and there is no prepared statement to switch to. A system-design problem, not a filter.