embedding-anisotropy-check

Embedding anisotropy check

Raw cosine between embeddings is rarely centred on zero. Most sentence encoders and most hidden-state layers place every vector near one shared direction, so two unrelated inputs can score 0.6, 0.9 or 0.99. A matched pair at 0.9 is then not evidence of anything by itself, and a gap of 0.01 between two conditions can be the whole signal or none of it. This skill makes every such comparison carry its own floor.

How an agent reaches this skill

The agent does not vectorise anything here. The skill applies at the point where the agent is about to report a cosine number, whoever produced the vectors:

• The agent wrote the evaluation. A request such as "write a retrieval eval for our index" or "check that the 4-bit export of this encoder still matches bf16" ends in a script that prints a mean cosine. Before the agent says "0.91, the export is fine", it runs the same model's unrelated pairs through the script and reports the floor beside the matched figure.
• The vectors already exist in the workspace. An embeddings.npy, a probe or adapter evaluated on saved hidden states, a vector-store dump. Asked "are these two layers encoding the same thing" or "did the fine-tune move the representations", the agent loads the arrays and runs the check before answering.
• A claim is being written. The agent is drafting a README, a PR description or a results table that calls a similarity high, says two conditions agree, or calls a small difference a null. The number in that sentence is the trigger.

When to use this skill

• The same inputs under one model at two precisions (bf16 against 4-bit), in two runtimes, or at two nearby layers, which share one coordinate space.
• Query and passage, image and caption, or any other paired evaluation inside one embedding space.
• Any sentence claiming that a similarity is high, that two conditions agree, or that a small difference is a null.

Not for two independently trained models as they stand. A rotation of one model's space keeps its geometry and changes every cosine against the other model, and centring does not align the two bases. Fit an explicit map first, for example orthogonal Procrustes on a train split of paired rows, and compare the mapped rows; or compare each model's own within-space similarities instead.

Procedure

1. Save the compared rows as two .npy arrays of shape (n, d), row i of A paired with row i of B.

2. Choose where the means come from. The best source is a train split from the same domain, model and modality as the compared rows. A mean fitted on another domain mis-centres: it can leave part of the shared direction in place, remove too much of it, or add a direction of its own, and what it leaves then looks like an effect of whatever differs between the domains.

3. Run the bundled script:

   python scripts/anisotropy_check.py A.npy B.npy --fit-a trainA.npy --fit-b trainB.npy
   

   Pass --shared when A and B come from one model and one population, so that both sides are centred by one mean. Without --fit-a and --fit-b the means are fitted on the compared rows themselves, and the output's means field says, for each side, which rows its mean came from.

4. Report, side by side: the anisotropy of each side, matched against floor before centring, matched against floor after centring, and partner_ranked_first against its chance rate 1/n.

Reading the output

• anisotropy is the mean cosine between different rows of one side. For large n it is close to the squared length of the mean of the unit rows, so it measures how much the rows share one common direction, which is what centring removes. It does not detect every departure from isotropy: rows spread along one axis in both directions score near 0 and still give raw cosines of +1 and -1. A value near 0 says there is little within-side mean direction to remove, not that raw cosine can be read without its floor. Above about 0.3, rows within that side cluster strongly around a mean direction (the script adds a warning); check raw.floor to see how much of that direction is shared across sides, and report both the raw and centred matched-versus-floor readings (centring changes the scoring function and does not replace evaluating an index that searches by raw cosine).
• raw.floor is what an unrelated pair scores in this comparison. A matched cosine means something only through its distance from this floor.
• centred.gap_over_floor_sd and centred.cohens_d say whether matched pairs separate from unrelated ones at all. When unrelated pairs all score the same, the floor has no spread and gap_over_floor_sd is null: the gap cannot be expressed in floor standard deviations, and the plain gap is the figure to read.
• partner_ranked_first is the retrieval view of the same pairs. Centring can move a ranking either way, so report it both ways rather than assuming it transfers.

Rules that go with the numbers

• Average the signed difference across seeds or items, then take its absolute value. Averaging absolute differences is biased upward, and the bias is greatest when the gap is small relative to its noise.
• Give a gap with its spread and the smallest effect the test could have detected. An underpowered comparison is not a refutation, and the report should say which of the two it is.
• Put the floor from the same comparison beside every result. A floor copied from another run, model or domain does not correct this one.

Example

bge-small-en-v1.5 on the first 100 documents of mteb/scifact's corpus, each split into its first and second half at the middle sentence boundary, with the halves of one abstract as the matched pair and one mean shared by both sides (--shared). scripts/example_scifact_halves.py rebuilds it: it pins the corpus at revision cf10ab6 and the model at 5c38ec7, states the sentence split, and needs datasets and sentence-transformers besides numpy. With datasets 5.0.1, sentence-transformers 6.0.1, torch 2.13.0 and numpy 2.5.1 it gives (rounded to four decimal places):

	matched	floor (unrelated halves)	gap over floor sd
raw cosine	0.8636	0.6024	4.38
centred cosine	0.6487	-0.0131	5.87

The anisotropy is 0.5956 and 0.6148, and here the two sides share one space, so before centring an unrelated half already scores 70% of what the matching half scores. Both spaces separate matched from unrelated halves (4.38 floor sd raw and 5.87 centred) and put the partner first for most abstracts (97 and 98 of 100), while centring removes the 0.6024 baseline offset.