Sentence embedding models turn a sentence into one vector so that millions of texts can be searched quickly. Many of them behave like word counters. "The dog bit the chef" and "The chef bit the dog" land almost on top of each other, while a paraphrase written in different words lands far away. We propose a specific explanation and a cheap fix. The explanation is that the encoder's per-token states still know who did what to whom, and the loss happens in the final averaging step, which is blind to order. The fix replaces that average with a small, fixed number of ordered slots. Each slot is a learned attention readout meant to capture one participant role. Two sentences are compared slot-to-slot, in correspondence, and never "best match anywhere." Concatenated, the slots are still one fixed-length vector. Standard nearest-neighbor search still works, and the storage cost of multi-vector retrieval never arises.
This matters because the two existing escape routes each give up something important. Multi-vector and token-matching methods keep more information, but they multiply index size and latency. Their "take the best match among vectors" scoring is itself order-invariant, so it plausibly keeps the word-counting bias. Hard-negative fine-tuning of ordinary single vectors works inside its training domain but is reported not to transfer across compositional phenomena. If role information really survives until pooling, then the pooling operator is the right place to intervene, and it can be tested cleanly.
Core Claim
We claim that, for off-the-shelf contrastive bi-encoders, most of the failure on high-overlap, meaning-changing pairs (role swaps, argument reordering) comes from commutative pooling, not from the token representations. We further claim that a pooling operator that is order-sensitive in role space, though not in surface position, recovers this distinction. Such an operator stays invariant to benign rewrites such as active-to-passive. It should also generalize to perturbation types it was not trained on better than single-vector hard-negative fine-tuning trained on the same data at the same dimension.
The operator is simple. Let $h_1,\dots,h_n$ be the encoder's token states. Let $q_1,\dots,q_k$ be $k$ learned slot queries. Slot $i$ is $u_i = \mathrm{norm}\big(\sum_t \mathrm{softmax}_t(q_i^\top W h_t)\, h_t\big)$, where $W$ is a learned projection. The similarity between sentences $a$ and $b$ is
$$s(a,b) = \sum_{i=1}^{k} \langle u_i(a), u_i(b) \rangle,$$
which is exactly the inner product of the concatenated vectors $[u_1;\dots;u_k]$. It is therefore compatible with any single-vector nearest-neighbor index. The contrast with multi-view retrieval is the whole point. There, the score is $\max_{i,j}\langle u_i(a), u_j(b)\rangle$ or a sum of per-token maxima. Swapping which content sits in which vector leaves that score unchanged, so it cannot separate "A bit B" from "B bit A" if the two sentences' slots merely trade places.
Where This Sits in Prior Work
Three papers motivate the diagnosis that the loss happens at pooling. Going Beyond Sentence Embeddings matches unpooled token states with significance and uniqueness weights. It lifts Mirror-BERT by 12.7% on STS, which suggests the token states carry information that pooling discards. However, it evaluates only on standard STS and itself flags word order as future work. Early compositional work, A Comparison of Vector-based Representations for Semantic Composition, shows that additive composition is commutative and therefore syntax-blind. Mean pooling inherits exactly this property. On the Difference of BERT-style and CLIP-style Text Encoders and ColorSwap argue that contrastive training with random negatives rewards keyword matching. This is the competing explanation: the encoder itself was trained into word counting. Our first experiment is designed to separate the two accounts.
Two papers supply methods we reuse. Multi-View Document Representation Learning introduced learned "viewer" tokens with a uniformity loss to keep them from collapsing. We borrow the anti-collapse loss but change the scoring from max-over-viewers to slot-wise correspondence, and we target within-sentence roles rather than document topics. Sentence Smith generates controlled edits from meaning graphs: role swap, negation, antonym, hypernym, underspecification. We use its edit families as the axes of held-out generalization.
Two papers constrain the design. ParaBART strips syntax out of the semantic vector with an adversarial discriminator. That buys invariance to rephrasing but, by construction, cannot tell role-swapped sentences apart. Our slots must be invariant to surface position yet sensitive to role, which is why training pairs passive rewrites as positives against role swaps as negatives. AoE attributes high-overlap failures to cosine saturation and fixes the metric in complex space. If AoE's account is right, a metric fix on a single vector should match our slots, so AoE is a required competitor. Finally, Aiming beyond the Obvious shows that aggregate scores are dominated by "obvious" pairs where overlap and label agree. We therefore report results on its non-obvious subsets rather than on raw averages.
Research Program
The first stage is a localization study on frozen encoders: all-MiniLM-L6-v2, Sentence-BERT, a SimCSE-style model, sentence-t5-large, and an NLI-tuned model. For each encoder we train small linear probes on role-swap pairs drawn from PAWS and Sentence Smith. One probe reads the token states and must decide which of two entity mentions is the agent. A second probe reads the mean-pooled vector and must decide which of a swapped pair is the original. If the token-state probe is near ceiling while the pooled probe is near chance, the pooling account holds for that encoder. If both fail, the encoder never encoded role, and ordered slots on a frozen encoder should not help. That outcome is itself a result: it would point back to the training objective as the cause, as the CLIP-style analysis suggests. We report the gap per encoder rather than assume it.
The second stage trains ordered slot pooling. We use $k \in \{2,4,8\}$ and keep the total dimension fixed at the base model's $D$, so each slot has $D/k$ dimensions. We train two variants: the encoder frozen, and the encoder unfrozen. Training data is general NLI plus PAWS training pairs. Positives are entailment pairs and passive or reordered paraphrases. Negatives are in-batch examples plus role-swapped non-paraphrases. We add the multi-view uniformity loss across slots so that different slots stay distinct. Crucially, training uses only word-order and role manipulations. Negation and antonym edits are never shown during training and serve as held-out generalization tests.
The third stage is analysis. We examine what each slot attends to, for example whether slot 1 consistently tracks the agent across active and passive voice. We also test whether slot structure survives on sentences with more than two arguments.
Baselines
The naive reference point is the unmodified mean-pooled bi-encoder, evaluated with cosine similarity. On SemAntoNeg, the SemAntoNeg study reports all-MiniLM-L6-v2 at 1.9% accuracy against a 33.3% random baseline. That figure gives the floor the method must clear on a benchmark it never trained for.
The strongest competitor on our core question is the same encoder with mean pooling, fine-tuned on exactly the same NLI, PAWS, and role-swap data, with the same hard negatives, the same number of steps, and the same hyperparameter search budget. We run it at dimension $D$ and also at $kD$, so that any gain cannot be attributed to extra dimensions. This baseline isolates the key ingredient, slot correspondence, from the data. Beating it means clearly higher accuracy on the held-out edit families (negation and antonym in Sentence Smith, and SemAntoNeg) with no loss on in-domain role swaps or STS. Matching it only on role swaps would count as a failure of the central claim.
For the SemAntoNeg comparison specifically, the strongest reported system is NLI-tuned models at up to 93.6% accuracy, per the SemAntoNeg study. We initialize from such a model and ask whether slots preserve that accuracy while adding role sensitivity.
The key ablation is ordered slots scored with maximum-over-slots instead of correspondence, with everything else identical. This single change turns the method back into an order-invariant multi-view scorer. If it performs as well as correspondence scoring, our mechanism story is wrong.
We also run the MVR-style multi-view scorer and the token-level matching method of Going Beyond Sentence Embeddings as post-hoc layers on the same encoder. For the latter, the file reports Spearman of roughly 80 or higher on STS-B. We run AoE-style angle optimization on a single vector, trained on the same data. A cross-encoder fine-tuned with PAWS augmentation serves as an accuracy ceiling rather than a competitor; PAWS reports BERT reaching 85% on its challenge set after adding 12,000 PAWS examples.
Why not just…?
Why not just fine-tune the single vector on hard negatives? This is the real threat. It may well close the in-domain role-swap gap on its own. The evidence against it is indirect. ColorSwap raised CLIP's group score from 11.67% to 63% with 1,400 targeted examples, yet the gain did not carry over to Winoground. That pattern is consistent with the model memorizing a pattern rather than gaining representational room. Our budget-matched baseline settles the question directly. If it equals ordered slots on held-out edit families, we will report that pooling was not the bottleneck and drop the claim.
Why not just use a cross-encoder? Cross-encoders see both sentences together, but Sentence-BERT reports about 65 hours to find the most similar pair among 10,000 sentences, against 5 seconds with pooled vectors and a nearest-neighbor index. PAWS also shows that cross-encoders drop to 33–40% on scrambled pairs without targeted data. We include one as a ceiling. The question we care about is how much of the gap a searchable single vector can close.
Why not just use multi-vector retrieval such as MVR or token matching? These methods store $k$ or $n$ vectors per text, which multiplies index size and latency. MVR reports 25ms per query against 320ms for ColBERT, both slower than single-vector search. More importantly, their best-match scoring is invariant to which vector holds which content. We predict they keep much of the role-swap blindness, and the max-over-slots ablation tests that prediction head to head.
Why not just use a larger embedding model? Sentence Smith reports that sentence-t5-large significantly outperforms E5-Mistral-7B on its GPTP edits. Its scores correlate only 0.54 with MTEB rankings, so scale alone does not appear to fix fine-grained role distinctions. We include E5-Mistral-7B as a reference in the evaluation.
Proposed Experiments and Decisive Checks
We evaluate on five sources. The first is the Sentence Smith foil set, scored by triple accuracy per edit type; the file reports no triple-accuracy numbers, so we measure all baselines ourselves. The second is SemAntoNeg accuracy. The third is PAWS-Wiki as a bi-encoder pair-classification task, keeping PAWS-QQP out of training so it can serve as a transfer test. The fourth is the jumbling and antonym tests from ALIGN-SIM. The fifth is STS-B and the STS 2012–2016 sets, reported on the non-obvious subsets defined by Aiming beyond the Obvious, to check that invariance to paraphrase is not lost. As a retrieval sanity check, we run Natural Questions retrieval top-5 accuracy with the concatenated slot vector in a standard index. MVR's reported 76.2% there is a reference point, not a target.
Three outcomes decide the proposal. First, the localization probe must show role information in token states that is absent from pooled vectors for most encoders; otherwise the premise fails. Second, correspondence scoring must beat max-over-slots on role swaps; otherwise the mechanism is not what we claim. Third, ordered slots must beat budget-matched hard-negative fine-tuning on held-out negation and antonym edits and on PAWS-QQP, while matching it on STS non-obvious positives. If all three hold, pooling is a repairable part of the word-counting problem, and a cheap single-vector repair exists.
References
- Going Beyond Sentence Embeddings: A Token-Level Matching Algorithm for Calculating Semantic Textual Similarity
- A Comparison of Vector-based Representations for Semantic Composition
- On the Difference of BERT-style and CLIP-style Text Encoders
- ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation
- Multi-View Document Representation Learning for Open-Domain Dense Retrieval
- Sentence Smith: Controllable Edits for Evaluating Text Embeddings
- Disentangling Semantics and Syntax in Sentence Embeddings with Pre-trained Language Models
- AoE: Angle-optimized Embeddings for Semantic Textual Similarity
- Aiming beyond the Obvious: Identifying Non-Obvious Cases in Semantic Similarity Datasets
- It Is Not Easy To Detect Paraphrases: Analysing Semantic Similarity With Antonyms and Negation Using the New SemAntoNeg Benchmark
- PAWS: Paraphrase Adversaries from Word Scrambling
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
- ALIGN-SIM: A Task-Free Test Bed for Evaluating and Interpreting Sentence Embeddings through Semantic Similarity Alignment