RoFormer: Rotary Position Embedding (RoPE)

Su et al., 2021

Position as a rotation, not an addition. Explained with a clock, because that is what it is.

Download rope_from_scratch.py — everything below runs from it. NumPy only.

I'd been serving LLaMA-family models for a year before I could say what rope_theta in the config actually did. I knew the 2017 paper adds a position vector to every token. I knew LLaMA doesn't. What nobody told me is that the replacement is smaller than the thing it replaced: turn each query and key by an angle, and the dot product forgets where the sentence started.

Position is a clock hand

The attention page ends with + pe. You know this:

x = embed(token) + pe[pos]     # 2017: position is a vector you ADD
q = x @ W_q
k = x @ W_k
score = q @ k                  # position baked in before the lookup

RoPE takes position out of the embedding and puts it into the lookup itself. Project first. Then turn: 

q = embed(token) @ W_q         # no position yet
k = embed(token) @ W_k
q = turn(q, by=pos * theta)    # 2021: position is an ANGLE you rotate by
k = turn(k, by=pos * theta)
score = q @ k                  # only sees the angle BETWEEN q and k

That's the paper. RoPE multiplies; it does not add. The payoff is school trigonometry: the dot product of two turned vectors depends only on the angle between them. Turn q by 3θ and k by 7θ, or by 13θ and 17θ — same score. Relative position for free, zero parameters.

Picture a clock with d/2 hands. Each pair of dimensions in q is one hand, position is how far it has turned, and attention sees exactly one thing: the angle between q's hand and k's hand. One hand, from the script: 

one hand, 30 deg per position, starting at [1. 0.]
  pos 0: turned   0.0 deg -> [1. 0.]   length 1.0000
  pos 1: turned  30.0 deg -> [0.866 0.5  ]   length 1.0000
  pos 2: turned  60.0 deg -> [0.5   0.866]   length 1.0000
  pos 3: turned  90.0 deg -> [0. 1.]   length 1.0000   ← never changes
TermMeansClock analogy
Pair i = dims (2i, 2i+1)One 2D slice of qOne hand
θiRadians pair i turns per positionHow fast that hand ticks
m θiPair i's angle at position mWhere the hand points
(n − m) θiThe only thing the score seesThe angle between two hands

Why this beat adding a vector

Adding leaks. Expand (q + pe[m]) · (k + pe[n]) and the cross terms q·pe[n] and pe[m]·k carry absolute position, so "the cat sat" scores one way at the start of a document and another way in the middle. The relative fixes before RoPE bolted a learned bias-per-gap table onto the score matrix: parameters, memory, and nothing to attach to in the linear-attention variants that never build that matrix. RoPE has no table and no parameters, is relative by construction, and drops into any dot-product attention. That is why LLaMA, PaLM, GPT-NeoX, Mistral and most models since use it.

The one formula

qm = R(m) q,  kn = R(n) k  ⇒  qm · kn = qT R(m)T R(n) k = qT R(n − m) k

q = Wqxm and k = Wkxn are the ordinary projections; R(m) is the rotation. Read it as pipeline stages:

StepShapeWhat it does
θi = 10000−2i/d(d/2,)One tick rate per pair. Pair 0 turns 1 rad per position; the last pair ≈ 1/10000
m θi(d/2,)How far each hand has turned at position m
R(m)(d, d), block-diagonald/2 independent 2×2 rotations. Never built as a matrix — it's a cos/sin multiply
R(m) q(d,)q with position baked in. Same length, new direction
qm · knscalarEquals qT R(n − m) k. Only the gap survives

The middle step is the whole proof. Rotations compose by adding angles and transposing one reverses it, so R(m)T R(n) = R(n − m). That identity is the paper; the rest is bookkeeping.

Fast hands, slow hands

Every pair gets its own tick rate, spanning four orders of magnitude. With d = 8 the rates are round numbers, so that's what the script prints: 

d = 8, so 4 hands. theta_i = 10000^(-2i/d) = [1.    0.1   0.01  0.001]

  at position 10, each hand has turned:
    pair 0:  10.0000 rad =   572.96 deg = 1.5915 turns   ← already lapped
    pair 1:   1.0000 rad =    57.30 deg = 0.1592 turns
    pair 2:   0.1000 rad =     5.73 deg = 0.0159 turns
    pair 3:   0.0100 rad =     0.57 deg = 0.0016 turns   ← barely moved

Pair 0 is the second hand: it resolves neighbours, but laps roughly every six positions and alone would alias. Pair 3 is the hour hand: useless for neighbours, but it tells position 10 from position 1000. The interview question is "why multiple frequencies?"; the answer is a clock with only second hands. The break table has the numbers.

The score only sees the gap

The paper's claim, so it gets the most careful print. Same q, same k, at positions (3, 7) and then at (13, 17): gap 4 both times. Then the 2017 add scheme, same frequencies.

Don't take my word for it. Here's the actual output:

RoPE, rotate:      q@3  . k@7  = -1.261507812773
                   q@13 . k@17 = -1.261507812773
                   difference  = 2.2e-16   <- float noise

2017, add:         q@3  . k@7  = 3.650608771116
                   q@13 . k@17 = 3.830134199073
                   difference  = 1.8e-01   <- absolute position leaks

unrotated q . k             = -0.468002152973
gap 0, e.g. q@5 . k@5       = -0.468002152973   <- identical

The last two lines are the sanity check: at gap 0 the rotation cancels and you get the plain dot product back. Shift the sentence by ten and RoPE does not notice. The add scheme moves by 1.8e-01 — not rounding, a different score.

RoPE multiplies. The 2017 scheme adds. Same frequencies, same sines and cosines, different operation — and that is why one leaks absolute position and the other cannot.

Far apart scores lower — on average

The paper's second claim is "long-term decay": the further apart two tokens, the smaller their dot product tends to be. It's the clock again: at gap 0 every hand lines up; as the gap grows the fast hands scatter and cancel, and only the slow hands still agree. The script sets q = k = all ones so the answer is just a sum of cosines: 

q = k = all ones, d = 64. dot(q at 0, k at gap):
  gap  0:   64.000
  gap  1:   61.834
  gap 16:   38.746
  gap 24:   39.446   ← went up
  gap 64:   27.812

mean dot over gap buckets (the trend):
  gaps  0- 7:   52.767
  gaps  8-15:   42.032
  gaps 16-23:   38.059
  gaps 24-31:   35.912
  gaps 32-39:   34.069
  gaps 40-47:   34.060
  gaps 48-55:   29.433
  gaps 56-63:   33.098

not monotonic: the dot RISES at 22 of the 64 steps, e.g.
  gap  5 ->  6: 47.008 -> 47.119

Be honest about what this is. From 64 down to about 28 over 64 positions is a real trend, and the paper's figure (an upper bound on this same sum) has the same shape. But it rises at 22 of 64 steps and the last bucket is higher than the one before. RoPE gives the model a tendency to look nearby, not a rule. A trained model attends far away whenever it needs to.

Q and K only. Never V.

The property lives in the score, so the rotation goes on q and k, right after W_q and W_k and before the dot product. V is the content being blended and is left alone. The test: five tokens at positions 0–4, then the same five at 100–104. With RoPE on q and k only the output must not change at all. Then rotate v as well: 

same 5 tokens at positions 0-4, then at 100-104
  CORRECT (q, k only):  weights max|diff| = 2.2e-16
                        output  max|diff| = 6.7e-16   <- shift-invariant
  BROKEN (v too):       output  max|diff| = 2.0167   <- moves with the sentence
                        vs correct output = 2.0803
  token 0, correct: [ 0.564  0.904  0.593 -0.455  0.815 -0.611  0.502 -0.447]
  token 0, v too:   [-0.437 -1.176  0.68  -0.335  0.827 -0.585  0.503 -0.445]

With v rotated the weights are still fine — same 2.2e-16 — but each token's output now depends on where the sentence started, precisely what RoPE exists to remove. Rotating after the softmax is worse: the weights were computed with no position at all (max|diff| from plain attention: 0.0000), so you are back to the attention page's "cat ate food" equals "food ate cat" problem, with a scramble on top.

Change the base, slow the clock

Every rate is 10000−2i/d. Change 10000 to 100000 and the fast hands barely notice (pair 0 is 1 rad per position for any base) while the slow hands slow right down. The script uses d = 128, LLaMA's per-head size, and looks at position 4096: 

base   10000: slowest hand theta = 1.155e-04 rad/pos
              at pos 4096: 0.4730 rad = 27.10 deg = 0.0753 turns
base  100000: slowest hand theta = 1.197e-05 rad/pos
              at pos 4096: 0.0490 rad =  2.81 deg = 0.0078 turns

slow hand, base 100000: reaches the base-10000 pos-4096 angle at pos 39513

This is why context extension is possible. A model trained to 4096 tokens has only ever seen its slowest hand between 0 and 27 degrees; ask for position 8192 with the same base and that hand points somewhere it never has, and quality falls off a cliff. Raise the base and the slow hands are stretched so 8192 lands inside the range training covered, while the fast hands that handle local structure are left almost untouched. That is the NTK-aware trick, and why rope_theta in LLaMA 3's config is 500000 rather than 10000. Position interpolation feeds position/8 instead (the script prints that too), but it slows every hand equally, fast ones included, which is why that paper fine-tunes afterwards. Both are one-line changes to rope_freqs. That's as deep as this page goes.

The code

Pure NumPy. No torch, no autograd, no training — just dot products, so you can watch the angle between two hands. Every function under 10 lines. The script also has rope_half (the half-split pairing), sinusoidal_pe (the 2017 add scheme) and the attention page's softmax; they're short and not repeated here.

Download the full script — 39 lines of core functions plus seven runnable demos. Then: 

python3 rope_from_scratch.py
import numpy as np

# STAGE 1 -- the tick rates. One per PAIR of dims, so d/2 of them:
#     theta_i = base^(-2i/d),   i = 0 .. d/2 - 1
# Pair 0 turns 1 radian per position (the fast hand); the last pair about
# 1/base (the slow hand). Same frequencies as the 2017 sinusoidal encoding.
# RoPE reuses them and does something different with them.
def rope_freqs(d, base=10000.0):
    i = np.arange(d // 2)
    return base ** (-2.0 * i / d)

# STAGE 2 -- THE WHOLE PAPER. Turn each 2D pair of dims by its own angle:
#     [x1']   [cos a  -sin a] [x1]
#     [x2'] = [sin a   cos a] [x2]
# Pair (0,1) gets angles[0], pair (2,3) gets angles[1], and so on. That is
# the paper's block-diagonal matrix R, never materialised: just a cos/sin
# multiply. No parameters. Nothing learned.
def rotate_pairs(x, angles):
    """x: (d,), angles: (d/2,). Pair i = dims (2i, 2i+1), turned by angles[i]."""
    c, s = np.cos(angles), np.sin(angles)
    x1, x2 = x[0::2], x[1::2]
    out = np.empty_like(x)
    out[0::2] = x1 * c - x2 * s
    out[1::2] = x1 * s + x2 * c
    return out

def rope(x, pos, base=10000.0):
    """Rotary embedding of vector x at integer position pos. Same shape out."""
    return rotate_pairs(x, pos * rope_freqs(x.shape[-1], base))

# STAGE 3 -- attention with RoPE. The attention page's function plus two
# lines: turn each row of Q and K by its own position, right after the
# projections and before the dot product. V is left alone.
def attention_with_rope(Q, K, V, positions, base=10000.0):
    Qr = np.stack([rope(q, m, base) for q, m in zip(Q, positions)])
    Kr = np.stack([rope(k, n, base) for k, n in zip(K, positions)])
    scores = Qr @ Kr.T / np.sqrt(Q.shape[-1])
    weights = softmax(scores, axis=-1)
    return weights @ V, weights

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living, and every row below is already wired into a demo.

Break thisWatch for
In attention_with_rope, rotate V as well: V = np.stack([rope(v, m) for v, m in zip(V, positions)])Demo 4. The same five tokens at 0–4 and at 100–104 now give outputs that differ by 2.0167. Correct version: 6.7e-16.
Add instead of turn. In demo 2 replace rope(q, 3) with q + sinusoidal_pe(3, d), and likewise for kThe gap-4 score moves by 1.8e-01 when the sentence shifts by ten. The relative property is gone; you have rebuilt the 2017 scheme.
Mix pairing conventions: rope(q, 3) @ rope_half(k, 7) (demo 6 does this on its last line)11.62921 at (3, 7) but 0.55906 at (13, 17). Either convention alone gives identical numbers under the shift. Hugging Face LLaMA uses rope_half; Meta's code uses adjacent pairs; the weight converter permutes Wq, Wk to match.
One tick rate for every pair: make rope_freqs return np.full(d // 2, 2 * np.pi / 8) (demo 7 does this locally)Gap 1, gap 9 and gap 17 all score 2.242033. Positions alias every 8 tokens; the model cannot tell one token back from nine.
Rotate after the softmax: leave Q, K alone and apply rope to the rows of weights @ VDemo 4, second block. The attention weights are identical to plain no-position attention, max|diff| 0.0000. Order-blind again, with the outputs scrambled by absolute position for good measure.
Then read the paper — but only the background section on absolute versus relative embeddings, then the 2D case and the general form in the method section, and the long-term decay figure now that you've printed the curve yourself. Skip linear attention and the experiments. You'll find the paper is just the comments you've already read.

On the maths anxiety

This paper needs less maths than the attention one. A 2×2 rotation matrix is four numbers made of one cos and one sin. "Rotations compose by adding angles" is the only fact used, and you have known it since you first turned a dial. The dot product as similarity, from the attention page, does the rest.

The paper writes the rotation as a complex number, q · eimθ. That is the same 2×2 rotation in shorter notation; read every eimθ as "turn by mθ" and you lose nothing. The one part you can genuinely skip is the Abel-transform bound in the decay section: it proves the trend from demo 3 and adds no mechanism.

What matters for the job: the rotation goes on q and k, not v; know which pairing convention your checkpoint uses before you port it; and rope_theta and rope_scaling in a config are the two knobs from the last section. None of that is research maths.

Where to go next

This page gave you one thing: RoPE as a clock, in code you can break. Two other pieces cover what I skipped, each better at its angle than I'd be.

Order I'd take them in: Fleetwood → EleutherAI → the paper. Fleetwood makes the design feel inevitable, EleutherAI makes it concrete, the paper is then a formality. Then the LLaMA page is where this gets used.