Machine learning papers, translated into code you can break on purpose.
For engineers who ship models, not for researchers.
def attention(query, keys, values):
scores = [dot(query, k) for k in keys]
weights = softmax(scores)
return sum(w * v for w, v in zip(weights, values))That's the beating heart of Attention Is All You Need. Three lines. Every paper here gets the same treatment: the idea as a function you could set a breakpoint in, real output from running it, and a short list of things to break so you can watch what fails.
ai-ml/transformers/
-
Byte-Pair Encoding: the tokeniser you inherited
Train BPE in a for loop, then watch why the model can't count letters and why the same sentence costs several times more in Tamil.
-
Attention Is All You Need
Attention is a dictionary lookup with fuzzy keys — everything else is plumbing. Runnable NumPy, four demos that prove it by breaking it.
-
RoFormer: Rotary Position Embedding
Rotate q and k instead of adding a position vector, and the dot product only sees distance. Every claim is a printed number.
-
BERT: Deep Bidirectional Transformers
Remove the causal mask, hide 15% of the tokens, guess them. Then bolt a head on per task. A tiny masked model, trained live.
-
GPT: Generative Pre-trained Transformers
Decoder-only, next token, then scale. The generation loop is a while loop. A character-level GPT trained in your terminal.
-
An Image Is Worth 16×16 Words (ViT)
The whole paper is a reshape: cut the image into patches and it's a sentence. Shapes printed at every step.
-
LLaMA: Open and Efficient Foundation Models
Not a new idea, a recipe: four deltas on the 2017 transformer and far more tokens than the scaling laws said. Parameter arithmetic included.
ai-ml/scaling/
-
Chinchilla: how big a model should you train?
Fixed compute budget: spend it on a bigger model or more data? Roughly twenty tokens per parameter, swept and plotted.
-
Mixture of Experts: the Sparsely-Gated Layer
A load balancer in front of eight small networks. Eight times the parameters, twice the compute. Hot shards and dropped tokens, as in production.
-
DeepSeek-V3: the recipe, and the interconnect that shaped it
What you build when your GPUs have the compute but not the bandwidth. Every choice traces to one halved interconnect.
ai-ml/inference/
-
The GPU, for people who only ever call .cuda()
One number, arithmetic intensity, predicts which of your operations will be slow before you profile anything.
-
KV caching and the generation loop
Why inference is affordable. The causal mask makes keys and values immutable, so you memoise them. Includes the cache-size arithmetic that decides what you can serve.
-
Grouped-Query Attention: one set of keys for several heads
The default in every modern open model. One repeat operation, and getting it wrong produces plausible garbage with no error.
-
Multi-head Latent Attention: the KV cache, compressed
Cache one small latent instead of every key and value, and fold the decompression into the query projection so it costs nothing.
-
FlashAttention: attention was never compute-bound
The n² cost is memory traffic, not arithmetic. Tile it, keep a running softmax, never store the score matrix.
-
Quantisation: what fits on your GPU
Divide, round, store. LLM.int8, GPTQ, AWQ and QLoRA's NF4 are four arguments about which numbers you may be sloppy with.
-
Speculative decoding: guess, then check
A cheap model guesses ahead, the big one verifies in a single pass. Branch prediction for tokens, with the output provably unchanged.
-
How a serving engine actually works
Continuous batching and paged memory. Your inference server is an operating system, and this is its scheduler and allocator.
ai-ml/fine-tuning/
-
LoRA: Low-Rank Adaptation
A fine-tune is a diff on the weights, and the diff is low-rank. Store the patch, not the repo. Merge it for free at inference.
-
Parameter-Efficient Fine-Tuning: the family
One question five ways: where do you insert the small trainable thing? Adapters, prefixes, prompts, (IA)³ and LoRA, with parameter counts.
ai-ml/alignment/
-
Reinforcement Learning from Human Feedback
You can't write the loss for "helpful", but you can ask which of two answers is better. Reward hacking, reproduced on purpose.
-
InstructGPT: Following Instructions with Human Feedback
The three-stage recipe that turned a completion engine into an assistant. Each stage's gain in one table. This is ChatGPT's recipe.
-
Direct Preference Optimization
The RLHF objective has an exact answer, and rearranging it makes the reward model disappear. Preference tuning becomes one classifier loss.
-
GRPO: reinforcement learning without a critic
PPO needs a second model to guess the baseline. Sample a group of answers and use their own mean instead. The algorithm behind R1.
-
DeepSeek-R1: reasoning nobody taught it
Reward only correct answers and long reasoning appears on its own, because thinking longer was the cheapest way to be right.
ai-ml/generative/
-
Auto-Encoding Variational Bayes (VAE)
An autoencoder whose bottleneck is a distribution. The reparameterisation trick is moving the random number outside the function.
-
Generative Adversarial Nets
A forger and a detective train each other. Mode collapse and vanishing gradients, triggered deliberately on a toy problem.
-
Denoising Diffusion Probabilistic Models
Learn to undo a noise-adding for-loop one step at a time. The forward process is a formula; the model is a one-step undo.
ai-ml/retrieval/
-
Sentence-BERT and CLIP: making distance mean something
Where embeddings come from. A model trained for something else gives a useless distance, measured and then fixed with contrastive loss.
-
Retrieval-Augmented Generation
An open-book exam. The retriever is the index, the prompt is the page you slide across. The failures are retrieval bugs, not model bugs.
ai-ml/prompting/
-
Chain-of-Thought and ReAct
A model gets fixed compute per token. Thinking out loud is how it buys more, and ReAct is how it buys facts.
-
Tool calling, MCP and RAG: three answers to different questions
Compared as rivals, but they solve different problems. Retrieve a fact, take an action, or standardise the wiring.
ai-ml/production/
-
Temperature, top-k and top-p: the three knobs on the API
Not three settings for one thing. Temperature reshapes the whole distribution, then k and p crop it, and the order matters.
-
Constrained decoding: making the model return valid JSON
Stop retrying until it parses. Mask the tokens that cannot continue a valid document, and invalid output becomes unrepresentable.
-
Hallucination and grounding: the model was never trying to be right
Nothing in training rewarded being correct. Why a fluent wrong answer is the objective working, and what grounding actually buys.
-
Evaluation: the part everyone skips
How big a golden set has to be before a difference means anything, and what a biased judge does to your numbers.
-
Prompt injection: why the model cannot tell your instructions from theirs
Instructions and data share one channel, and there is no prepared statement to switch to. A system-design problem, not a filter.
Who writes this
Srinivasan Nagaraja Rao. AI engineer. I ship code and models for a living, and I'm learning this field in public — including the parts where the maths scares me.
I'm not a researcher. I'm a programmer. When I read a machine learning paper, I don't want the proof — I want to know what it would look like as a function I could put a breakpoint in. So that's how I write these up.
Why "for dummies"
Because most paper explainers are written by people who are comfortable with the maths, aimed at people heading toward research. That's a real audience. It just isn't mine.
Mine is the engineer who has shipped models, done the MLOps, and still feels like a fraud when someone opens a paper. I'm writing these from inside that exact confusion, while I still remember what was confusing.
The method: translate the idea into code you already understand, then deliberately break it and watch what fails. You debug for a living. Reverse-engineering from broken behaviour is faster than re-reading section 3.2 for the fifth time.