Search engines that store one vector per document lose detail: a single fixed-length point cannot hold every fact in a long passage. Multi-vector retrievers address this by keeping one vector per token and scoring a query by letting each query token find its best-matching document token. These systems are expensive, so the leading way to make them affordable is averaging. Similar token vectors are grouped and each group is replaced by its mean. We argue that this averaging step quietly turns a retriever that could tell "the chef bit the dog" from "the dog bit the chef" into one that cannot. It merges the subject-"chef" and object-"chef" tokens into one centroid, because both are dominated by the identity of the word "chef." We propose to measure this effect directly and then to fix it with a compression objective that is told which distinctions it must not average away.
The two-sentence version is this. Token-level retrievers are not bags of words by construction: each token vector is contextualized, so it can encode that "chef" is the agent. Our claim is that bag-of-words behavior is reintroduced by two separable causes. The first is contrastive training that never shows the model a role-swapped negative. The second is centroid-based compression that groups tokens by word identity. Each cause can be measured and corrected without paying for a cross-encoder.
Core Claim
The retrieval literature currently treats late interaction as a soft bag of words. An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Models shows that ColBERT-style scores come mostly from term co-occurrence and high-IDF keywords. The scoring rule, a sum over query tokens of the maximum similarity to any document token, is indeed indifferent to the order of query tokens. That analysis motivates our work, but it conflates the scoring rule with the representations it scores. Probing work shows the encoder does carry role information. When classifying grammatical role, BERT doesn't care about word order... except when it matters finds that roles in non-prototypical sentences like "the onion chopped the chef" are at chance in early layers but become resolvable in middle and later layers. Multimodal evidence shows that training data decides whether that capacity is used. In ColorSwap, CLIP scores a 12% group score on attribute-swap pairs, but rises to 63% after fine-tuning on 1,400 swap examples.
Our hypothesis is that in text retrieval, role information survives in uncompressed token vectors once training includes role-swapped negatives. Compression then removes it. Neither CRISP: Clustering Multi-Vector Representations for Denoising and Pruning nor ColBERTv2 has been evaluated on inputs where word order is the only difference. CRISP mean-pools K-means clusters of token vectors during training. ColBERTv2 stores each token as a corpus-level centroid plus a 1–2 bit residual. Classic composition results give a precise prior for what averaging does to role structure. In Experimental Support for a Categorical Compositional Distributional Model of Meaning, additive composition falls to Spearman $\rho = 0.05$ on subject–verb–object disambiguation, against 0.21 for a structure-aware model. We expect the analogous collapse inside clusters. In ColBERTv2's case, we expect role information to sit almost entirely in the residual that quantization discards.
Research Program
The work has three parts: a test set, a diagnosis, and a fix.
The test set. We will build a role-swap retrieval test, because none exists among the benchmarks in the file. It has two halves. The natural half comes from MS MARCO and BEIR passages. We parse each passage, swap the subject and object (or a modifier between two nouns) in one clause, and pair the original query with both versions. Annotators then filter swaps that are ungrammatical or still compatible with the query. The synthetic half extends the attribute-combination design of On the Theoretical Limitations of Embedding-Based Retrieval to directed relations, such as "X recommended Y to Z." Each entity pair appears in many role configurations, so a corpus can contain dozens of reversed distractors per gold document. That benchmark's original LIMIT design tests unordered attribute sets only, and our extension is what makes word order matter. We report two numbers. The first is minimal-pair accuracy: how often the gold document scores above its swapped twin. The second is Recall@k in a corpus seeded with reversed distractors.
The diagnosis. This is a two-by-two study. One factor is training: standard contrastive training with mined hard negatives, versus the same training with role-swapped negatives added at a fixed fraction. The other factor is the stored representation: full token vectors, ColBERTv2-style centroid plus residual at 0, 1, 2 and full bits, and CRISP-style trained cluster means at several compression ratios. We also fit linear probes for grammatical role on three things: the full token vector, its centroid, and its residual. This locates where role information lives.
The fix. If the diagnosis confirms the hypothesis, we add one ingredient to CRISP-style training, which we call contrast-preserving pooling. During each forward pass, the role-swapped twin of a positive document is encoded and pooled with the same clustering procedure. An auxiliary margin loss then requires the pooled positive to outscore the pooled twin by a margin $m$:
$$ \mathcal{L}_{\text{role}} = \max\big(0,\; m - S(Q, \bar{D}) + S(Q, \bar{D}^{\text{swap}})\big) $$
Here $S$ is the max-similarity late-interaction score, $\bar{D}$ is the set of cluster means for the document, and $\bar{D}^{\text{swap}}$ is the set for its role-swapped twin. The gradient pushes role-distinct tokens of the same word into different clusters, because merging them would make the two pooled sets nearly identical. The vector budget stays fixed, so storage and latency match CRISP exactly.
Baselines
The naive reference point is BM25. It assigns identical scores to a passage and its role-swapped twin by construction, so its minimal-pair accuracy is chance. This floor shows the test isolates order rather than vocabulary.
The single-vector dense dual encoder of Sparse, Dense, and Attentional Representations for Text Retrieval shows what the fixed-length bottleneck alone does. That paper reports MRR@10 of 0.302 on MS MARCO for the BERT dual encoder and 0.334 for its multi-vector ME-BERT. We will include a mean-pooled variant as the averaged-embedding reference.
The strongest existing compressed systems are ColBERTv2 and CRISP. ColBERTv2 reports MRR@10 of 39.7 on MS MARCO with a 16–25 GiB index (ColBERTv2). CRISP reports BEIR NDCG@10 of 54.5 against 54.3 unpruned at 2.9× fewer vectors (CRISP). The strongest late-interaction method that explicitly targets token relations is TRIAL. It adds a learned pairwise token-relation score and reports 46.3 nDCG@10 on MS MARCO and 51.09 average on BEIR, at quadratic cost in sequence length. There is no reported result found on Lacuna for any of these systems on role-swap or word-order minimal-pair retrieval, so we will produce their numbers ourselves.
All learned systems share a backbone, training data, number of training steps, and hyperparameter search budget. The CRISP-family comparisons are matched on vectors per document. A cross-encoder reranker over the top 100 serves as the upper reference.
The ablation that isolates our ingredient is CRISP trained with role-swapped negatives added to the ordinary contrastive loss, but without the pooled-twin margin loss. That run separates "better data" from "compression told what to preserve."
Beating the field means two things. On the role-swap test, contrast-preserving pooling must match the minimal-pair accuracy of uncompressed ColBERT trained with swapped negatives at CRISP's vector budget. On BEIR, it must stay within half a point of CRISP's 54.5 NDCG@10.
Why not just…?
Why not just add role-swapped hard negatives and keep CRISP as is? This is the most serious alternative, since ColorSwap shows data alone moves CLIP from about 12% to 63%. The ablation above tests it directly. If swapped negatives restore role sensitivity under ordinary cluster averaging, we will report that the problem is data, not pooling, and drop the margin loss. Our prediction is the opposite. The ordinary loss only shapes token vectors before pooling, and K-means then merges tokens that share a word regardless of context. The centroid-versus-residual probes will show which prediction holds.
Why not just rerank with a cross-encoder? Cross-attention is known to handle this kind of binding, as BLIP's 87% on ColorSwap shows. On natural queries with one swapped twin per gold document, a top-100 reranker probably fixes most errors, and we will say so. It fails when reversed distractors outnumber the rerank depth and push the gold document out of the candidate list. Our synthetic relational corpus is built to create exactly that regime. We will plot Recall@k against the number of distractors so the crossover is visible rather than asserted.
Why not just skip compression, or keep more residual bits? That is the right answer if storage is free. The bit sweep over ColBERTv2 residuals tells practitioners exactly how many bits role information costs. If two bits suffice, residual quantization is already enough and our method is unnecessary for ColBERTv2-style indexes. It may still be needed for CRISP-style aggressive pooling, where the 11× compression point leaves no residual at all.
Why not just use TRIAL? TRIAL models token pairs and could capture argument direction. Its pairwise term grows quadratically with length, and it was evaluated on phrase coherence rather than role reversal. We will test it on the role-swap set at matched training. Our method's advantage, if any, is that it adds no cost at query time.
Proposed Experiments and Decisive Checks
The first decisive check settles the location of role information. Uncompressed ColBERT trained with swapped negatives should reach high minimal-pair accuracy. Its ColBERTv2-compressed and CRISP-pooled versions should fall toward the BM25 floor as compression grows. Role probes should succeed on residuals and fail on centroids. If uncompressed tokens are themselves at chance, the hypothesis is wrong: the problem lies in the encoder or the scoring rule, and the program pivots toward TRIAL-like relational scoring.
The second check compares contrast-preserving pooling against the swapped-negatives ablation at 2.9× and 11× compression. Our method should hold minimal-pair accuracy while the ablation degrades with compression. If the two are tied, the data-only explanation wins.
The third check confirms that no ordinary retrieval ability is lost. We will report MS MARCO MRR@10, BEIR NDCG@10 and LoTTE nDCG@10 against the published CRISP, ColBERTv2 and TRIAL figures. We will also report relational-corpus Recall@k curves against a cross-encoder at rerank depths of 100 and 1,000. Finally, we will cross-check against the ABNIRML perturbation probes, which find neural rankers sensitive to word order but inconsistent. The aim is to confirm that gains reflect role sensitivity rather than a generic penalty on edited text.
References
- An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Models
- When classifying grammatical role, BERT doesn't care about word order... except when it matters
- ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation
- CRISP: Clustering Multi-Vector Representations for Denoising and Pruning
- ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
- Experimental Support for a Categorical Compositional Distributional Model of Meaning
- On the Theoretical Limitations of Embedding-Based Retrieval
- Sparse, Dense, and Attentional Representations for Text Retrieval
- TRIAL: Token Relations and Importance Aware Late-interaction for Accurate Text Retrieval
- ABNIRML: Analyzing the Behavior of Neural IR Models