Sentence-BERT and CLIP: making distance mean something

Reimers & Gurevych, 2019 · Radford et al., 2021

Two papers, one idea: you have to train for the property you want to measure.

Download embeddings_from_scratch.py — everything below runs from it. NumPy, plus a tiny CPU PyTorch model for the training step. About 11 seconds.

The RAG page cheats. It says so out loud: the retriever there is hashed bag-of-words, because the real thing needed a page of its own. This is that page. Where the vectors come from.

I assumed for an embarrassingly long time that any decent language model hands you embeddings for free. Run BERT, take the CLS vector, cosine away. It doesn't work, and the reason it doesn't work is the finding in Sentence-BERT: raw BERT output was never trained to put similar things close together, so it doesn't. The vectors all crowd into a narrow cone and everything is 0.9 similar to everything.

A hash function versus a similarity index

You've written both. This one:

h = hashlib.sha256(doc.encode()).hexdigest()   # a thing -> a number

Perfectly good identity. Deterministic, fast, collision-resistant. And completely useless for "find me documents like this one", because two nearly identical documents hash to two unrelated values. Distance in hash space means nothing. That's the whole design goal — a good hash destroys structure.

Then there's the other kind:

v = embed(doc)                     # a thing -> a vector
near = sorted(corpus, key=lambda d: -cosine(v, embed(d)))[:5]

Same signature. Thing in, number out. But this one has to preserve structure, and preserving structure is not something that happens by accident. It happens because somebody trained it to.

Both papers do the same thing in the same way. Sentence-BERT takes BERT, runs two copies of it in a siamese setup with a pooling step, and fine-tunes so cosine similarity between the two output vectors matches whether the sentences actually mean the same thing. CLIP takes an image encoder and a text encoder, runs them on a large set of image-caption pairs, and trains so a photo lands near its own caption and far from everyone else's.

TermMeansProgrammer's analogy
EmbeddingA vector where direction encodes meaningA sort key, not an ID
Bi-encoderEncode each item once, compare vectorsBuild an index, then look up
Cross-encoderFeed both items in together, score the pairA full table scan with a join predicate
AnisotropyAll vectors crammed into a narrow coneA hash with a terrible distribution
In-batch negativeEvery other item in the batch, used as a wrong answerFree test data you already loaded

Why the bi-encoder mattered: it made search affordable

Before Sentence-BERT, the good way to compare two sentences with BERT was a cross-encoder: concatenate them, push both through the model, read a score off the top. Accurate. Also quadratic. Comparing every pair in a collection of n sentences is n²/2 forward passes through a large transformer, and you cannot cache anything, because the score only exists for the pair.

A bi-encoder embeds each sentence once — n forward passes, done offline — and then comparison is a dot product. That's the difference between a job you run overnight and a job you run never.

The trade is real and it's the interview question. A cross-encoder is usually more accurate, because it lets the two texts attend to each other. A bi-encoder is enormously faster. Production systems use both: bi-encoder to fetch a hundred candidates from millions, cross-encoder to rerank the hundred. That's what a reranker is, and it's the piece the RAG page leaves as a stub.

The one formula

Not an architecture. A loss. This is InfoNCE, and it is what both papers are actually contributing:

L = −log [ exp(sim(ai, bi) / τ) ÷ Σj exp(sim(ai, bj) / τ) ]

Squint and it's cross-entropy, because that's exactly what it is. Read it as pipeline stages:

StepShapeWhat it does
a, b(B, d) eachA batch of B matched pairs, L2-normalised, one row per item
a @ b.T(B, B)The score matrix. The diagonal is the B correct pairings; everything off it is a wrong answer you got free
/ τscalarTemperature. How sharply the softmax discriminates
softmaxrow-wiseRow i becomes "how confident am I that bj is the match for ai"
cross_entropylabels = 0..B−1The label for row i is i. That's the whole trick

You never write a single label. The batch index is the label. Load 64 matched pairs and you have handed the model 64 questions with 63 wrong answers each — 4032 negatives, for free.

The cone: why raw CLS vectors aren't embeddings

My script's stand-in for "a network trained for something else" is a randomly initialised encoder. Not identical to BERT, but it has the property that matters — real weights, real non-linearity, never trained for this distance. Measure the average cosine between unrelated items, before and after training:

Don't take my word for it. Here's the actual output:

untrained encoder, mean pairwise cosine: +0.6781
                   min / max cosine:     -0.1540 / +0.9814

trained   encoder, mean pairwise cosine: +0.0041
                   min / max cosine:     -0.9113 / +0.9316

Before training, two items with nothing in common sit at 0.68, and plenty of pairs are above 0.98. Your similarity threshold has nowhere to go: the whole distribution is jammed into the top of the range. After training the cosines use the range they're supposed to, and the average unrelated pair is at zero, which is what "unrelated" ought to look like.

This is measurable, so measure it. If you inherit an embedding service and want to know in one minute whether it's any good, take 200 random items, compute the mean off-diagonal cosine, and see if it's near zero.

The objective, running

This is the one place I use torch. The mechanism — similarity, retrieval, the anisotropy measurement — is NumPy, but the paper's idea is a training objective, and hand-rolled backprop would bury it under plumbing. 

batch 64, temperature 0.07. Chance loss = ln(64) = 4.1589

  step    1   loss 5.6448
  step   50   loss 0.6859
  step  100   loss 0.3321
  step  200   loss 0.3079
  step  300   loss 0.3596
  step  400   loss 0.2470

Step 1 is worse than guessing, which is right — the encoders are random. Fifty steps in, the loss is well under ln(64) and the model has learned the only thing it was asked to learn: put matching pairs closer together than the other 63 items in the batch.

Retrieval before and after: the whole argument, in two numbers

Loss curves are easy to fool yourself with. The number that settles it is retrieval accuracy: query with item i from one side, score all 200 items on the other, and check whether i comes out on top. 

chance (1/200)                 : 0.0050
untrained encoders, top-1      : 0.0050
trained encoders,   top-1      : 0.8650

Same architecture. Same data. Same cosine function. The untrained encoders score exactly at chance — one hit in two hundred. Cosine distance over vectors from a model that wasn't trained for it is not a weak signal. It is no signal.

That is Sentence-BERT's finding, restated. Their experiments found mean pooling over BERT's token outputs generally beat using the CLS vector — but the real point is that neither is any good until you fine-tune for the similarity objective. Pooling choice is a detail. Training for the property is the paper.

Temperature: both ends of the dial are broken

τ divides the similarities before the softmax. Small τ makes the distribution sharp, so the loss obsesses over the single hardest negative. Large τ flattens it, so the loss barely distinguishes the right answer from the wrong ones. Sweeping it, retraining from scratch each time: 

   tau  final loss    top-1   sim std  mean top score
 0.005      0.5941   0.6950    0.0914          0.8183
 0.020      0.3950   0.8500    0.2666          0.9030
 0.070      0.3506   0.8650    0.3451          0.8836
 0.300      1.9828   0.8650    0.3559          0.9080
 1.000      3.3512   0.8150    0.3896          0.9265
10.000      4.0698   0.6200    0.4861          0.9458

sim std is how spread out the similarity distribution is. At τ=0.005 it collapses to 0.09 — the space folds up, because chasing one hard negative per step is a terrible training signal — and accuracy drops to 0.695. At τ=10 the loss barely moves off ln(64), so nothing separates, and accuracy drops to 0.62. The usable band in the middle is wide and flat, and both edges are cliffs.

CLIP's answer to picking a number is to not pick one: the temperature is a learned parameter, optimised along with the weights.

Normalisation, or the longest vector wins

Cosine is a dot product divided by both lengths. That division is not decoration. Here are the same trained embeddings, with the indexed items given varying magnitudes — the way real documents have varying lengths: 

cosine (normalised),   top-1 : 0.8650
raw dot product,       top-1 : 0.3100
raw dot product, distinct items ever returned: 66 of 200
raw dot product, most-returned item index    : 197

Two thirds of the corpus becomes unreachable. Not because those items are irrelevant — because they're short. This is the identical bug the RAG page demonstrates on unnormalised document scores, and it's the same fix: divide by the norm, or store the vectors already normalised so you can't forget.

Two towers, one space

CLIP's structural idea in one sentence: two encoders that share nothing, reading two different input types, producing vectors in the same space, trained only by the loss that pulls matched pairs together.

In my script, modality A is 32 features and modality B is 24. They're generated from a shared hidden cause plus noise. They stand in for images and captions — they are not images and not text, and I'm not going to pretend otherwise. What they preserve is the only structural thing that matters: two unrelated input formats, one thing underneath. 

 query (A)   best match (B)   its cosine   runner-up   correct?
         0                0       0.7954      0.7021       True
         1                1       0.9028      0.8699       True
         2                2       0.8894      0.7815       True
         3              140       0.8045      0.7582      False
         4              179       0.8307      0.7893      False
         5                5       0.7668      0.7065       True

A->B top-1: 0.8650     B->A top-1: 0.8450

Note rows 3 and 4 — this toy is noisy enough to fail sometimes, which is honest. Note also that it works in both directions, because the loss is symmetric: A retrieves B, and B retrieves A.

This is where zero-shot classification comes from. Once images and text share a space, you classify by embedding the image, then embedding a text prompt for each candidate class, and taking the nearest. No classifier head, no training on your label set. The classes are just strings you typed. CLIP's paper is largely about how far that gets you across a lot of benchmarks — go read their numbers rather than mine.

The code

NumPy for the mechanism: cosine similarity, the anisotropy measurement, the retrieval evaluation. A tiny PyTorch model, CPU only, for the one part that needs gradients. Every function under 15 lines.

Download the full script — about 100 lines of mechanism plus seven runnable demos. Then: 

python3 embeddings_from_scratch.py
import numpy as np
import torch
import torch.nn as nn


# STAGE 1 -- cosine similarity, and the two things people get wrong about it.
# cos(u, v) = (u . v) / (|u| |v|). Dividing by the lengths is what makes it
# a similarity and not a dot product. Skip the division and the longest
# vector wins every time -- the same bug the RAG page demonstrates on
# unnormalised document scores.
def l2_normalise(X, eps=1e-9):
    return X / (np.linalg.norm(X, axis=-1, keepdims=True) + eps)


def cosine_matrix(A, B):
    """(n, d) x (m, d) -> (n, m) of cosines. Normalise, then one matmul."""
    return l2_normalise(A) @ l2_normalise(B).T


def mean_pairwise_cosine(X):
    """Anisotropy, measured. Average cosine between distinct random vectors.

    If the vectors are spread over the sphere this sits near 0.0. If they all
    crowd into a narrow cone it sits near 1.0, and 'close' stops meaning
    anything -- everything is close to everything.
    """
    C = cosine_matrix(X, X)
    n = C.shape[0]
    off_diagonal = C[~np.eye(n, dtype=bool)]
    return float(off_diagonal.mean())


# STAGE 2 -- retrieval evaluation. The only number that settles an argument.
# Query with row i of A, score every row of B, and check whether row i comes
# out on top. Chance level is 1/n.
def top1_accuracy(A, B, normalise=True):
    scores = cosine_matrix(A, B) if normalise else A @ B.T
    return float((scores.argmax(axis=1) == np.arange(len(A))).mean())


# STAGE 3 -- the two towers. One encoder per input type, one shared output
# space. CLIP's image encoder and text encoder; Sentence-BERT's two copies of
# BERT with tied weights and a pooling step on top.
# Nothing here is clever. Two small MLPs. What matters is the objective they
# are trained under, not the architecture.
class Tower(nn.Module):
    def __init__(self, d_in, d_out=D_EMB, hidden=64):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(d_in, hidden), nn.ReLU(), nn.Linear(hidden, d_out)
        )

    def forward(self, x):
        # L2-normalise on the way out, so the space we train is the space we
        # search: cosine similarity becomes a plain dot product.
        return nn.functional.normalize(self.net(x), dim=-1)


# STAGE 4 -- InfoNCE with in-batch negatives. The whole contribution, in six
# lines.
# For a batch of B matched pairs, build the (B, B) matrix of similarities.
# The diagonal holds the B correct pairings. Every off-diagonal cell is a
# negative you got for free -- no negative sampling, no mining, just the rest
# of the batch. Then it is a B-way classification problem, and the label of
# row i is i.
# The temperature divides the scores before the softmax. Small temperature =
# sharp, the loss cares only about the hardest negative. Large temperature =
# flat, the loss barely distinguishes anything.
def info_nce_loss(emb_a, emb_b, temperature):
    logits = emb_a @ emb_b.T / temperature      # (B, B) -- diagonal is correct
    labels = torch.arange(len(emb_a))
    loss_a = nn.functional.cross_entropy(logits, labels)      # A retrieves B
    loss_b = nn.functional.cross_entropy(logits.T, labels)    # B retrieves A
    return (loss_a + loss_b) / 2                # symmetric, as in CLIP

Now break it

This is the part that actually makes it stick. Reading won't do it. You debug for a living — reverse-engineering from broken behaviour is your native mode, and every row below is one edit away.

Break thisWatch for
Skip training: score with untrained_towers() instead of train_towers() Top-1 retrieval falls from 0.8650 to 0.0050 — exactly chance on 200 items. The raw-CLS-vector mistake, measured.
Pass normalise=False to top1_accuracy 0.8650 → 0.3100, and only 66 of 200 items are ever returned. Magnitude beats meaning. Same failure as the RAG page's unnormalised scores.
train_towers(temperature=10.0) Final loss sits at 4.0698, barely under ln(64)=4.1589. The softmax is flat, the gradient can't tell right from wrong, accuracy 0.6200.
train_towers(temperature=0.005) The other cliff. Similarity std collapses to 0.0914 and accuracy drops to 0.6950 — the space folds up chasing one hard negative per step.
train_towers(batch_size=2) One negative per query instead of 63. Accuracy 0.3050. Batch size is the difficulty of the task here.
Query with run A's encoder against run B's index: train_towers(seed=7) for the second tower only Top-1 0.0000 — below chance. Same code, same data, different seed, unrelated spaces. Nothing throws.
That last row is the production incident. You upgrade your embedding model, re-embed half the corpus, and ship. No error, no alert, no exception — the vectors are still the right shape and the cosines still come back as plausible-looking floats. Retrieval quality just quietly dies for half your documents. Version your index by model, and re-embed all of it or none of it.
Then read the papers. For Sentence-BERT: the approach section (the pooling strategies and the siamese setup) and the ablation table where mean pooling is compared against CLS. Skip the per-dataset result tables unless you're chasing a specific benchmark. For CLIP: the approach section, including the pseudocode figure for the loss — it's about fifteen lines and it's the same six lines you've just run — and the zero-shot prompt-engineering discussion. Skip most of the evaluation, it's very long.

On the maths anxiety

There's less here than the notation suggests. A dot product is "how aligned are these two arrows". Dividing by both lengths turns it into cosine, so only direction counts. Softmax turns any row of numbers into percentages summing to one. Cross-entropy is "how surprised was the model by the right answer". You've now used all four.

InfoNCE looks fearsome written out with the sum in the denominator, but the denominator is just "every candidate in the batch" and the whole expression is cross_entropy(a @ b.T / tau, arange(B)). One line. If the formula and the line ever disagree in your head, trust the line.

What I did not do here, and what this page skips: why anisotropy arises in transformer representations in the first place (there's real literature on that, and it's more subtle than "training didn't ask for it"), the alignment-and-uniformity analysis of why contrastive losses work, and any actual text or images. This is the mechanism, not the phenomenon.

Where to go next

This page gave you one thing: why an untrained distance is worthless, and the six-line loss that fixes it, in code you can break. Everything else — real sentences, real images, real benchmarks — is elsewhere, and these four are better at it than I would be.

Order I'd take them in: Sentence-BERT first (it's the shortest and it's the one whose problem you've just felt), then the Sentence Transformers docs to build something, then CLIP, then Weng when you want the theory underneath. Weng first will just be symbols.