DeepSeek-V3: the recipe, and the interconnect that shaped it

DeepSeek-AI, 2024

Almost every choice in this model is downstream of one number on a spec sheet.

Download deepseek-v3_from_scratch.py — everything below runs from it. NumPy only, and nothing in it is a benchmark: it counts bytes and simulates schedules.

I read this report the way everyone did — looking for the trick. There isn't one. There are about eight, and they all point the same direction, and I couldn't see the direction until I noticed which GPUs they were on.

US export controls meant DeepSeek trained on H800s. Same chip as an H100 for arithmetic. Roughly half the NVLink — the chip-to-chip link inside a server. For a mixture-of-experts model that is the cruellest possible cut, because routing tokens to experts is an all-to-all, and all-to-all is exactly what a narrow link punishes.

You've built this system before

You have a service that fans a request out to backends and merges the replies. On a fast local network you write it the obvious way: 

def handle(req):
    picks = router.pick(req, k=8)              # 8 backends, wherever they live
    return merge([call(b, req) for b in picks])   # ship the payload to each

Then you move it to a link a third the speed, and it falls over. Not because the backends got slower — they didn't — but because the payload now spends longer on the wire than in the handler. You already know the three fixes, because you've shipped all of them: 

def handle(req):
    picks = router.pick(req, k=8, max_racks=4)  # 1. cap the fan-out
    payload = compress(req)                     # 2. shrink every byte
    fut = send_async(picks, payload)            # 3. overlap the wire with work
    local = do_other_work()
    return merge(fut.result(), local)

That is DeepSeek-V3's engineering half, in three lines. Cap the fan-out is node-limited routing. Shrink every byte is FP8. Overlap the wire with work is DualPipe. Everything else is detail about how far below the abstraction layer they were willing to go to make those three actually happen.

TermMeansOps analogy
All-to-allEvery GPU sends a slice of its tokens to every other GPU holding a chosen expert, then gets results backA shuffle stage. The bit of your MapReduce that actually costs money
NVLinkThe link between GPUs inside one server. 900 GB/s on H100, 400 GB/s on H800The rack-local network
InfiniBandThe link between servers. 400 Gbps = 50 GB/s per GPUCross-datacentre. Same request, ten times the latency budget
Expert parallelismExperts live on different GPUs, so tokens travel to reach themSharding by key. The shuffle is the price
BubbleA pipeline stage sitting idle waiting for its neighbourA worker blocked on IO with nothing queued behind it
PTXNVIDIA's assembly-level layer, one below CUDA CDropping to inline asm because the compiler won't do what you want

The one formula

Not a loss function. The formula that matters here is the one that decides whether your GPUs are computing or waiting:

tcomm / tcompute = (bytes / bandwidth) ÷ (FLOPs / throughput)

Read it as pipeline stages:

StepFor one MoE layerWhat it does
FLOPsk × 3 × 2 × dmodel × dffn per tokenTop-8 SwiGLU experts. Three matrices each, two flops per multiply-add
bytesnodes × dmodel × (1 + 2) on IB, k × dmodel × (1 + 2) on NVLinkOne copy per destination node out, results back. FP8 out, BF16 back
throughput1979 TFLOPS FP8Identical on H100 and H800. This is the number export controls did not touch
bandwidth900 or 400 GB/sThe whole paper. This is the number they did touch
ratio < 1compute-boundYour tensor cores set the clock. Make the model bigger
ratio > 1comm-boundYour wire sets the clock. Every design decision changes

The constraint, counted

DeepSeek-V3 is 671B total parameters, 37B active per token, 61 layers, dmodel 7168, 256 routed experts plus one shared, top-8 routing, expert hidden size 2048. Take one GPU with a micro-batch of 4096 tokens through one MoE layer and count.

The bandwidth numbers below are published specifications, not something I measured, and the "times" are bytes divided by those specifications. This is the same kind of counting the FlashAttention page does for HBM traffic — a roofline sketch, not a profile.

Don't take my word for it. Here's the actual output:

tokens on this GPU        : 4096
d_model                   : 7168, top-8 routed experts, capped at 4 nodes
expert FLOPs per token    : 704.6 MFLOP
expert FLOPs, this batch  : 2.89 TFLOP
InfiniBand bytes  (out+in): 352.3 MB (4 node copies x (1B dispatch + 2B combine))
NVLink bytes      (out+in): 704.6 MB (8 expert copies inside the nodes)
arithmetic intensity of this layer:
  2,731 FLOP per byte moved
machine balance (FLOP per byte the link can feed, FP8 compute):
  H100 NVLink 900 GB/s: 2,199
  H800 NVLink 400 GB/s: 4,948
  InfiniBand   50 GB/s: 39,580

2,731 FLOP per byte is what this layer offers. 2,199 is what an H100's NVLink can feed. 4,948 is what an H800's can. The layer sits between them. That is the entire story, and here it is as time: 

  machine          compute   NVLink leg   comm/compute   verdict
  H100  900 GB/s    1.46ms      0.78ms          0.54x   COMPUTE-bound
  H800  400 GB/s    1.46ms      1.76ms          1.21x   COMM-bound

  same layer, same arithmetic, one number on a spec sheet halved,
  and the bottleneck moves from the tensor cores to the wire.
Nothing about the model changed. Same weights, same FLOPs, same batch. One spec-sheet number halved and the layer crossed from compute-bound to communication-bound. Every remaining section of this page is a response to that crossing.

And the cross-node leg, which export controls did not touch, is worse than either — which is why the node cap exists at all: 

the cross-node leg, which the export cut did NOT touch:
  H100: IB  7.05ms + NVLink  0.78ms = 5.37x compute   (9.0x the NVLink leg)
  H800: IB  7.05ms + NVLink  1.76ms = 6.04x compute   (4.0x the NVLink leg)

The report is explicit that it tuned the node cap so the InfiniBand and NVLink legs can be overlapped, citing an effective NVLink-to-IB bandwidth ratio of about 3.2:1. That implies an achieved NVLink figure nearer 160 GB/s than 400. Treat my 900 and 400 as ceilings nobody reaches; the shape of the table is the point, not the decimals.

Auxiliary-loss-free balancing: the neatest idea in the paper

A router that plays favourites is the hot-shard problem, and the MoE page fixes it the standard way: add an auxiliary loss that penalises uneven routing. It works. It also adds a gradient — one that is, by construction, not the task's gradient. You are asking the router to be balanced instead of being right.

V3 replaces it with something an SRE would recognise immediately: a thermostat.

selection_score = affinity + bias      # bias affects WHO IS PICKED
gate            = affinity             # and never the loss, so grad(bias) = 0

bias[expert] -= gamma * sign(load[expert] - mean_load)   # after each step

Overloaded expert, bias down. Starved expert, bias up. Fixed step, sign only. It moves the selection and leaves the gradient alone. My script sets up a toy world where token demand is Zipf-skewed — so the routing the model wants is genuinely imbalanced, and balance and quality really do fight. 

true expert demand (Zipf, the routing the model actually wants):
  [162  90  73  49  43  35  27  33]

max/min expert occupancy, by step (1.0 = perfectly balanced):
  step      none      aux-loss      bias (V3)
     0      11.2        11.2           11.2
    10       8.7         3.8            8.1
    20       6.6         3.2            2.0
    40       6.1         3.0            2.2
    59       6.1         3.0            2.1

final per-expert token counts (512 tokens, top-1):
  none    : [165  89  74  51  41  33  27  32]
  aux-loss: [121  92  71  56  48  41  42  41]
  bias    : [80 53 60 74 62 50 43 90]
  V3 bias vector: [-3.  -1.8  1.2  1.2  3.   3.6  3.6  3. ]

Both balance. Now the number that motivates the whole idea — the gradient the balancing mechanism contributes to the router's weights, and how well the router still picks the right expert once you take the bias back off: 

  gradient norm contributed by the balancing term:
    aux-loss : 0.434286   <- fights the routing the model wanted
    bias (V3): 0.000000   <- exactly zero, by construction
  router accuracy vs the true best expert (bias removed):
    no balancing : 0.9590
    aux-loss     : 0.8848
    bias (V3)    : 0.9590

Better balance and no accuracy loss. Turn the auxiliary weight up and you get the failure the paper is built to avoid — you buy balance you already had, with quality: 

BREAK IT: crank the auxiliary loss weight to alpha=6.0
  balance  max/min : 3.1  (aux 1.50: 3.0, bias: 2.1)
  router accuracy  : 0.7324  <- balanced, and worse
  balancing grad   : 1.323678

The thermostat has its own failure mode, and it's the one you'd guess:

BREAK IT: bias update rate gamma=8.0 instead of 0.6
  max/min occupancy, last 8 steps:
    [202.   81.8 224.  202.   63.6 153.   63.4 153. ]
   vs gamma=0.6: [2.  1.6 2.  1.6 2.  1.6 1.3 2.1]
  final bias vector: [144. 144. 144. 144. 144. 152. 144. 144.]
Step size too big, and it oscillates instead of settling. It's a controller. It has a gain, and you tune the gain. The report's own caveat is worth keeping: they also use a small sequence-wise balancing loss as a backstop, so "auxiliary-loss-free" means the primary mechanism is the bias, not that no balancing loss exists anywhere.

Node-limited routing: capping the fan-out

A token wants its top-8 experts wherever they live. Left alone it can touch eight different servers, and every extra server is another InfiniBand copy of the same 7168-wide vector across the slowest link in the machine. The cap: score each node by the sum of its experts' affinities, keep the best M nodes, and pick the top-8 only from those. V3 uses M = 4. 

256 experts over 32 nodes (8 experts per node), top-8, 4096 tokens

  uncapped: mean 6.652 nodes/token, max 8
  capped  : mean 3.748 nodes/token, max 4  (M=4)

  InfiniBand bytes, uncapped: 585.9 MB  (11.72 ms at 50 GB/s)
  InfiniBand bytes, capped  : 330.1 MB  (6.60 ms at 50 GB/s)
  saved: 43.7%

what the cap costs -- experts still reachable per token:
  uncapped : 256 of 256 (100.0%)
  capped   : 32 of 256 (12.5%)
  mean overlap with the uncapped choice: 3.62 of 8 experts

It is not free, and the script prints exactly how unfree: a capped token gets 3.62 of the 8 experts it would otherwise have chosen, and only an eighth of the expert pool is reachable at all. You are trading model freedom for wire time. Sweep the cap and watch the trade: 

  M=2: 2.000 nodes/token,    176.1 MB IB, overlap 2.57/8
  M=4: 3.748 nodes/token,    330.1 MB IB, overlap 3.62/8
  M=8: 5.330 nodes/token,    469.5 MB IB, overlap 5.00/8
  M=32: 6.652 nodes/token,    585.9 MB IB, overlap 8.00/8

M is not chosen to minimise traffic — M=2 would do that. It is chosen so the IB leg and the NVLink leg are close enough in size to hide underneath each other, given that 3.2:1 ratio. The cap and the overlap are one design, not two.

FP8: the other half of the byte reduction

Halving the fan-out halves the copies. Halving the payload halves what each copy carries. V3 trains the GEMMs in 8-bit floating point — which, if you've read the quantisation page, you'll recognise as the same argument one octave further down. Two formats exist and neither is good enough alone: 

format   exp  man     max finite   min normal   steps per octave
  e4m3    4    3            448    1.562e-02      8
  e5m2    5    2         57,344    6.104e-05      4

e4m3 has the precision and not the range. e5m2 has the range and not the precision. Cast real tensors straight in and watch which dies: 

direct cast, one tensor, no scaling -- relative L2 error:
  tensor                     e4m3        e5m2     e4m3 finite?
  weights   (N(0,0.02))    0.310877    0.051356     True
  activation (N(0,1))      0.027098    0.053512     True
  act+outlier (x1500)           inf    0.024086     False
  grads     (N(0,1e-5))    1.000000    1.000000     True

which tensor dies first: the one with the outlier.
  max |activation| = 1,500.0  vs e4m3 max 448
  values that became inf: 1
  gradients underflow e4m3's min normal: 4096 of 4096 -> 0

One value out of 4096 exceeded 448 and the tensor is gone. A single FP32 scale per tensor brings it back — but that scale is set by the loudest value, and the quiet ones pay for it. Smaller blocks, more scales, fewer dead values: 

  direct cast, no scale : inf   <- overflowed, the tensor is gone
  one FP32 scale        : 0.001151   finite again

4096 activations, N(0,1), with four values of 3000 dropped in.
Error measured on the 4092 ORDINARY values only:
  block   err on quiet values   dead (rounded to 0)   bits/value
   4096             0.031677                   343        8.008
    512             0.029651                   184        8.062
    128             0.026348                    38        8.250
     32             0.024253                    12        9.000

That is V3's fine-grained quantisation: 1×128 tiles for activations, 128×128 for weights. The format did not get better. The blocks got smaller. The report pairs it with FP32 accumulation and wider master weights and optimiser state, because the places 8 bits are unacceptable are exactly the places error compounds. What it buys: 

  one activation tile, 4096 tokens x 7168:
    BF16                    58.72 MB
    FP8 (1x128 scales)      30.28 MB
  all-to-all bytes/token: BF16 dispatch 344,064 B ->
                          FP8  dispatch 258,048 B (25.0% less)

Multi-token prediction: two targets for one forward pass

A standard language model gets one training signal per position: predict the next token, as on the GPT page. V3 adds a second head that predicts the token after that, sequentially, keeping the full causal chain. Same forward pass, twice the targets. 

micro-batch: 4096 positions
  MTP depth 0 (1 head): 4,096 prediction targets per forward pass  (1.0x)
  MTP depth 1 (2 heads): 8,192 prediction targets per forward pass  (2.0x)
  MTP depth 2 (3 heads): 12,288 prediction targets per forward pass  (3.0x)

what each head predicts, for the sentence positions 0..4:
  pos 0 (the  ) -> head 1: cat     head 2: ate
  pos 1 (cat  ) -> head 1: ate     head 2: the
  pos 2 (ate  ) -> head 1: the     head 2: food

V3 uses depth 1 — one extra head. It is a training objective; the extra head is dropped for ordinary inference. But you can keep it, and then it is a draft model that costs almost nothing, which is speculative decoding with the drafter already inside the target model. The report puts second-token acceptance at 85–90% and reports roughly 1.8× decoding throughput from using it that way. 

expected tokens accepted per decoding step = 1 + p(second accepted):
  acceptance 0.85  ->  1.85 tokens/step (1.85x, before the extra head's own cost)
  acceptance 0.90  ->  1.90 tokens/step (1.90x, before the extra head's own cost)

The 1 + p arithmetic is mine, not theirs, and it ignores verification cost. The 85–90% and the 1.8× are the report's.

Overlapping communication with compute

You cannot delete the all-to-all. You can make it happen while the GPU is doing something else — the async call you already write without thinking. My script simulates a 1F1B pipeline, 8 stages, 32 microbatches, and asks what the device is doing. This is a simulation of a schedule, not a measurement of DeepSeek's. 

pipeline: 8 stages, 32 microbatches, compute 1.0, all-to-all 0.8 per microbatch

  schedule            makespan   device busy   bubble
  no overlap             70.20        45.6%     54.4%
  comm hidden            39.00        82.1%     17.9%

  wall-clock penalty for not overlapping: 1.80x
  illustration only: a 56.7-day run with this shape becomes 102.1 days

The second column is the one worth staring at. Once communication is hidden, making it more expensive stops mattering until it exceeds compute: 

  comm    bubble (no overlap)   bubble (overlapped)   speedup
   0.2                31.6%                 17.9%       1.20x
   0.5                45.3%                 17.9%       1.50x
   0.8                54.4%                 17.9%       1.80x
   1.2                62.7%                 31.6%       1.83x
   2.0                72.6%                 59.0%       1.50x
That flat 17.9% is the design goal. Get the communication cost under the compute cost and it vanishes from the wall clock entirely. DualPipe goes further than my simulation — it runs the pipeline from both ends at once — but the part that matters here is the part I can show you: hide the wire behind the maths.

What "went below the abstraction layer" actually means

Three things in the report that most write-ups skip, and that are the reason the three fixes above were achievable rather than aspirational:

What they didWhy
Wrote PTX, NVIDIA's assembly-level layer below CUDA, for the communication kernelsThe dispatch/combine kernels had to be tuned by hand. Inline asm, because the compiler wouldn't do it
Reserved a fraction of each GPU's streaming multiprocessors for communicationComm and compute genuinely run at once. You budget the cores, the way you'd pin a thread pool
Avoided tensor parallelism entirelyIt is the most communication-hungry parallelism there is — it talks inside every layer. On a narrow link it's the first thing to go

Multi-head latent attention is theirs too, from DeepSeek-V2: it compresses the key-value cache into one low-rank latent vector per token, so long-context serving stops being a memory problem. It has its own page — multi-head latent attention, built on the byte counting from the KV cache page — so I've deliberately not re-derived it here.

The number everyone gets wrong

You will be asked about the $5.6M. Do the division yourself once and you'll never repeat it carelessly: 

  final pre-training GPU-hours (report) : 2,788,000 H800-hours
  assumed rental price (report's own)   : $2.00 per GPU-hour
  product                               : $5.576M
  on 2048 GPUs that is 56.7 days of wall clock
  tokens: 14.8e12 over 671e9 params, 37e9 active per token
  tokens per active parameter           : 400
  approx forward+backward FLOPs (6*N*D) : 3285.6 ZFLOP
  implied MFU against BF16 peak         : 33.1%
  against the FP8 peak they actually used: 16.5%
That figure is the final pre-training run only, at a rental price the report itself assumes. It excludes research, ablations, failed runs, the earlier DeepSeek models this one builds on, salaries, and the cluster. The report says so directly. "DeepSeek built a frontier model for $5.6M" is not what the report claims.

The 400 tokens per active parameter is worth a second look next to the scaling laws page — it is far past Chinchilla-optimal, which is what you do when you intend to serve the model rather than win a compute-optimal argument.

The code

Pure NumPy. No torch, no GPU, no training run. Everything is arithmetic: bytes counted, formats simulated, schedules stepped. Every function under 20 lines.

Download the full script — 718 lines plus 7 runnable demos. Then: 

python3 deepseek-v3_from_scratch.py
def moe_flops_per_token():
    """One token through top-8 SwiGLU experts. 3 matrices, 2 flops per MAC."""
    per_expert = 3 * 2 * D_MODEL * EXPERT_HIDDEN
    return TOPK_ROUTED * per_expert


def all_to_all_bytes_per_token(nodes_touched, dispatch_bytes, combine_bytes):
    """Bytes a single token pushes across each link type in one MoE layer.

    Dispatch: the hidden vector goes OUT to every node holding a chosen
    expert -- one InfiniBand copy per node, then NVLink fans it out to the
    GPUs inside that node. Combine: the expert outputs come back the same
    way. V3 dispatches in FP8 and combines in BF16, so the two directions
    are not the same size.
    """
    ib = nodes_touched * D_MODEL * (dispatch_bytes + combine_bytes)
    nvl = TOPK_ROUTED * D_MODEL * (dispatch_bytes + combine_bytes)
    return ib, nvl


def seconds(byte_count, bandwidth):
    """Bytes / bandwidth. This is the whole 'timing' model. No GPU was run."""
    return byte_count / bandwidth


def route(X, W_g, bias=None, k=1):
    """Affinity scores, then top-k selection. Bias shifts SELECTION only."""
    scores = X @ W_g                       # (n_tokens, n_experts)
    probs = softmax(scores)                # what the model believes
    sel_scores = scores if bias is None else scores + bias
    chosen = np.argsort(-sel_scores, axis=1)[:, :k]
    return probs, chosen


def bias_balance_step(bias, chosen, n_experts, gamma):
    """V3's replacement, in three lines. Auxiliary-loss-free balancing.

    Starved expert -> bias up. Hot expert -> bias down. Fixed step size,
    sign only, like a thermostat. Never touches the loss, so it contributes
    NO gradient to W_g -- the router keeps learning what it wanted to learn.
    """
    load = load_counts(chosen, n_experts)
    err = load - load.mean()                 # + means overloaded
    return bias - gamma * np.sign(err)


def node_limited_topk(scores, n_nodes, k, m_cap):
    """Restrict candidates to the m_cap best nodes, then take top-k."""
    n_tokens, n_experts = scores.shape
    node_id = node_of_expert(n_experts, n_nodes)
    per_node = n_experts // n_nodes
    node_score = scores.reshape(n_tokens, n_nodes, per_node).sum(axis=2)
    keep = np.argsort(-node_score, axis=1)[:, :m_cap]      # (n_tokens, m_cap)
    allowed = np.zeros((n_tokens, n_nodes), dtype=bool)
    np.put_along_axis(allowed, keep, True, axis=1)
    masked = np.where(allowed[:, node_id], scores, -np.inf)
    return np.argsort(-masked, axis=1)[:, :k]

fp8_cast, fp8_blockwise and pipeline_bubble are the same shape and live in the script — the FP8 grid is np.frexp, round the mantissa, overflow to inf; the pipeline is a dozen lines of nested loop over stages and microbatches.

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living — reverse-engineering from broken behaviour is your native mode, and every row below is one number in the script.

Break thisWatch for
In demo_2_balancing, set train("aux", alpha=6.0) as the main runBalance stops improving at max/min 3.1 while router accuracy falls 0.9590 → 0.7324. You bought nothing and paid for it.
In demo_2_balancing, set gamma=8.0 on the bias runOccupancy stops settling and oscillates: [202. 81.8 224. 202. 63.6 …] instead of [2. 1.6 2. 1.6 …]. The controller's gain is too high.
In demo_4_fp8, call fp8_cast(a, "e4m3") instead of fp8_blockwiseThe activation tensor with the 1500 outlier goes to inf — relative error inf, one value overflowed, whole tensor dead. Gradients at N(0,1e-5) die the other way: 4096 of 4096 flushed to zero.
In demo_3_node_limited_routing, set NODE_CAP_M = 32 (every node)Cross-node traffic climbs from 330.1 MB to 585.9 MB, 6.60 ms → 11.72 ms at 50 GB/s. Tokens reach 8.00/8 of their preferred experts, and the IB leg no longer fits under the NVLink leg.
In pipeline_bubble, pass overlap=FalseBubble 17.9% → 54.4%, makespan 39.00 → 70.20. A 56.7-day run becomes 102.1 days at that ratio.
In all_to_all_bytes_per_token, pass dispatch_bytes=2 (BF16, not FP8)All-to-all bytes per token go 258,048 → 344,064, a 33% increase on the link that was already the bottleneck.
Then read the report — the infrastructure sections are the good ones: the FP8 training framework, DualPipe and the communication kernels, and the auxiliary-loss-free balancing part of the architecture section. Skip the benchmark tables unless you're picking a model. For the cluster underneath it, the Fire-Flyer paper is the companion. You'll find the report is mostly the comments you've already read.

On the maths anxiety

There is almost no maths on this page, and that is not me being kind. The spine of DeepSeek-V3 is division: bytes over bandwidth, FLOPs over throughput, compare. If you have ever sized a queue, worked out whether a job is IO-bound, or argued about payload compression, you already own the reasoning.

The two places actual maths appears, you can take on trust. The auxiliary loss has a gradient and a bias term does not — that's a one-line consequence of what "in the loss" means, and my script prints the two norms rather than deriving them. FP8's range and precision come from counting exponent and mantissa bits, which is a spec-sheet lookup, not analysis.

What this page does not give you: the actual convergence behaviour of FP8 training, the real profile of their kernels, and anything about post-training. Those need a cluster, and no amount of NumPy substitutes for one. What it gives you is the ability to say why each choice exists, which is the interview question.

Where to go next

This page gave you one angle: the interconnect as the cause, everything else as effect. That deliberately underplays the model as a model. These three fill in the rest.

Order I'd take them in: Lambert first, because it takes ten minutes and stops you saying something wrong; then the V3 report's infrastructure section; then Fire-Flyer for the machine; then the scaling book when you want to do this arithmetic for your own deployment rather than read mine.