Dense retrievers turn a query and each document into a single vector and rank documents by vector similarity. They are known to behave like sophisticated word counters. Sentence encoders give more than 94% cosine similarity to "cats chase mice" and "mice chase cats" (How Well Do Text Embedding Models Understand Syntax?). Retrievers also lose two-thirds of their ranking quality on queries containing "not" (BoolQuestions). The standard fix is contrastive training on "hard negatives": wrong documents that look deceptively close to the right one. Applied to word-order and role errors, this fix has a documented cost. Improving sensitivity to who-did-what-to-whom lowers general semantic similarity scores, which the earlier paper calls a "see-saw." A second, opposite line of hard-negative work, LED, deliberately teaches retrievers to reward exact keyword overlap. Meanwhile the vocabulary-mismatch problem requires them to match documents that share no words at all with the query. In WikiQA, 20.3% of correct answers share zero content words with the question.
Our hypothesis is that the see-saw is not a capacity limit. It is a side effect of how the negatives are chosen. If every structural hard negative is a high-overlap wrong answer, the cheapest way for the encoder to reduce the loss is to distrust word overlap in general, not to read structure. We therefore propose overlap-balanced contrastive training. Each training group pairs a structural negative (same words, relations broken) with two kinds of positives: a high-overlap paraphrase (same words, structure changed but meaning kept) and a low-overlap paraphrase (different words, same meaning). Within each group, word overlap then carries no information about relevance, so the only way to lower the loss is to encode relational structure.
Core Claim
Define a document's lexical overlap with a query as content-word Jaccard overlap, excluding named entities. On a held-out controlled probe set, we measure how much a retriever's similarity score depends on overlap once true relevance is accounted for, by fitting
$$s(q,d) = \alpha + \beta \cdot \mathrm{ov}(q,d) + \gamma \cdot \mathrm{rel}(q,d) + \epsilon,$$
where $s$ is the retriever's similarity, $\mathrm{ov}$ is the overlap, $\mathrm{rel}\in\{0,1\}$ is gold relevance, and $\beta$ and $\gamma$ are the fitted overlap reliance and relevance sensitivity. We predict the following signatures. Lexical-negative training such as LED should push $\beta$ upward. Naive structural augmentation should push $\beta$ below zero, and this anti-overlap shift should be what costs it on STS-B and on high-overlap relevant documents. Overlap-balanced training should hold $\beta$ near zero while raising $\gamma$ on relational probes as much as naive augmentation does.
The claim is falsifiable on two fronts. Balanced training must remove the see-saw while matching naive augmentation's relational gains. The size of the see-saw across training variants should also track $|\beta|$. If it does not, the mechanism is wrong even if the method happens to work.
Research Program
The first component is a generator of structural negatives that produces grammatical text. Earlier perturbation studies shuffled n-grams and produced word salad (Out of Order), which mixes up two failures: the model ignoring syntax, and the model being confused by input unlike anything it was trained on. Instead, we follow the finding of Alleman et al. that encoders treat swaps of whole syntactic phrases very differently from swaps of arbitrary spans. We swap complete argument phrases around a predicate, using a dependency parser and a semantic role tagger. The tagger is the same kind of off-the-shelf tool that PAS+SBERT feeds into its architecture. We use it only to build training data, so it adds no cost at retrieval time.
For a training pair from MS MARCO, we locate the passage sentence with the highest overlap to the query. We swap its agent and patient phrases, or the source and destination in a PAWS-style itinerary swap (PAWS), and leave the rest of the passage untouched. For example, "Facebook acquired Instagram in 2012" becomes "Instagram acquired Facebook in 2012." The result is a negative with roughly 99% lexical overlap with the positive.
Some predicates are symmetric ("A met B"), so swapping their arguments does not change the meaning. To avoid creating false negatives, which the hard-negative literature names as the main risk, we drop swaps whose predicate appears on a symmetric-predicate list. We also drop swaps that an MNLI cross-encoder judges to entail the original in both directions.
The second component supplies the balancing positives. High-overlap positives are meaning-preserving structural edits, such as voice alternation and clause reordering, the same transformations as the Structural Heuristics dimension of the SR benchmark. Low-overlap positives are back-translated paraphrases of the answer sentence, following the PAWS construction. We keep only those whose non-entity content-word overlap with the original falls below a threshold, and we verify them with the same two-way entailment check.
Each training group then holds the query, the original passage, its two paraphrased positives, the structural negative, one BM25 lexical negative of the LED kind, and in-batch random negatives. The loss is a multi-positive contrastive loss over the group. Its one real hyperparameter is how many structural negatives each positive receives. This is the sensitivity LED reports for its own negative counts, and we will sweep it.
The third component is a diagnostic study of the starting encoder. Drop your Decoder shows that pre-training the retriever's summary vector to predict an unordered bag of words gives strong results (40.1 MRR@10 on MS MARCO). That objective explicitly rewards ignoring word order. We will start balanced fine-tuning from both a bag-of-words-pretrained checkpoint and a plain BERT checkpoint. The question is whether order-agnostic pre-training permanently limits how far $\gamma$ can rise on relational probes, or whether fine-tuning recovers it.
The fourth component is a retrieval benchmark that tests both pressures at once. Current benchmarks test either vocabulary gaps (BEIR, WikiQA) or high-overlap structural adversaries (PAWS, HANS, the SR benchmark), and the structural ones are only sentence-pair tasks. We build an adversarial corpus from MS MARCO dev and Natural Questions queries whose answers depend on a relation. For each relevant passage, we add its role-swapped twin to the corpus, so a structure-blind retriever ties the true passage with an impostor. We also add a low-overlap relevant paraphrase, so a retriever that defends itself by matching keywords loses recall.
We will report three things on this corpus. Pairwise twin accuracy is how often the true passage outranks its swapped twin. Impostor-at-10 is how often the twin appears in the top ten, analogous to BoolQuestions' NegRecall@10. Low-overlap recall is how often the paraphrased relevant passage is retrieved. A human-validated subset of about 1,000 queries serves as the test set.
Baselines
The naive reference point is the retriever fine-tuned on MS MARCO with BM25 and in-batch negatives only, using the same backbone, data, and number of steps as every other variant. It fixes the starting values of $\beta$ and $\gamma$ and the baseline retrieval quality.
The strongest existing method on standard retrieval is LED, which trains with high-BM25 irrelevant passages as negatives plus pairwise rank-consistency distillation from a lexicon-aware teacher. It reports 40.2 MRR@10 on MS MARCO Dev with ranker distillation (LED). It is the right competitor because it is the best hard-negative recipe in the file that handles lexical overlap explicitly. It is also the method whose push toward rewarding overlap is most likely to hurt relational probes. We will reproduce it on our backbone with its published negative counts and loss weight, and tune it with the same number of trials as our method. Beating it means matching its MS MARCO Dev MRR@10 within 0.5 points while clearly improving twin accuracy and impostor-at-10.
The strongest existing method on structure is naive targeted augmentation as in How Well Do Text Embedding Models Understand Syntax?. That approach adds 10,000 syntactically varied examples and raises Relational Understanding Spearman correlation from 24.9% to 60.1% on the SR benchmark, at a cost on STS-B. We run two versions. One uses our structural negatives without the balancing positives. The other uses its own LLM-generated data, in equal quantity to ours. Beating it means coming within two points of its relational score with no statistically significant drop on STS-B, MS MARCO Dev, or the BEIR average.
We also compare against BoolQuestions' Boolean contrastive continued training on its own NOT subset. On that subset, gte-Qwen2-7B-instruct falls from 52.04 MRR@10 on "ALL" to 17.16 on "NOT." The comparison tests whether structural balancing transfers to negation, which our generator never produces.
Two ablations isolate the key ingredient. The first removes the low-overlap positives and keeps the high-overlap ones. The second does the reverse. If only one kind of positive is needed, the overlap-decorrelation account is incomplete, and we will say which half of the balance does the work.
Why not just…?
Why not just re-rank with a cross-encoder? Cross-attention alone does not enforce structure. BERT cross-encoders trained on QQP drop from above 90% to 33–40% on PAWS (PAWS), and 75–90% of GLUE predictions survive 1-gram shuffling (Out of Order). A reranker also cannot recover a low-overlap relevant passage that the first stage never retrieved. We will still run BM25 plus a cross-encoder reranker on the adversarial corpus. If it wins on twin accuracy, our contribution narrows to recall at the first stage, and we will report it that way.
Why not just add the structural negatives and replay the original data to prevent forgetting? This is the most serious threat. The see-saw might simply be forgetting caused by a small, unusual augmentation set, in which case mixing the original MS MARCO data back in, or distilling ranks from the original model in the LED style, would fix it without any balancing. We include this baseline: naive structural negatives combined with replay at a 1:1 ratio, plus rank-consistency distillation from the step-zero model, under the same tuning budget. If it closes the STS-B and BEIR gap as well as balancing does, we drop the balancing claim. The $\beta$ diagnostic separates the two explanations. Forgetting predicts a general loss of quality with $\beta$ near zero, while our account predicts a negative $\beta$ that replay does not remove.
Why not just rewrite queries with an LLM, or decompose them? Rewriting can bridge vocabulary gaps. It does nothing, however, to stop a retriever from scoring "mice chase cats" as highly as "cats chase mice," because the impostor passage stays in the corpus with identical words. On logical queries, the decomposition baseline in BoolQuestions reduces false positives but substantially lowers MRR@10. We include an LLM-rewrite front end on our adversarial corpus to measure how much of the low-overlap-recall problem it solves on its own.
Proposed Experiments
All variants share one backbone and one MS MARCO training budget. We will also start from the bag-of-words-pretrained checkpoint to address the pre-training question. In-domain evaluation uses MS MARCO Dev MRR@10 and TREC DL 2019 and 2020. Vocabulary-gap evaluation uses the BEIR average and the zero-overlap WikiQA subset. Structure evaluation uses the SR benchmark's two dimensions, PAWS-Wiki and PAWS-QQP scored by a fixed similarity threshold, HANS non-entailment scored by threshold, and BoolQuestions NOT. General similarity is measured on STS-B. The new adversarial corpus supplies twin accuracy, impostor-at-10, and low-overlap recall.
The decisive check has three conditions, which we pre-register. First, balanced training reaches at least 90% of naive augmentation's gain in SR Relational Understanding. Second, it shows no significant drop on STS-B or on the BEIR average; significance is assessed by paired bootstrap over five seeds. Third, across all trained variants, the size of each variant's see-saw correlates with $|\beta|$. A secondary check compares the replay baseline with balanced training on $\beta$. A tertiary check asks whether the bag-of-words-pretrained start ends with lower relational $\gamma$ after identical fine-tuning, which would mean order-agnostic pre-training imposes a lasting cost. If the replay baseline matches balanced training on every metric and also yields $\beta$ near zero, the paper becomes a negative result: the see-saw is forgetting, not overlap bias, and we will report it as such.
References
- How Well Do Text Embedding Models Understand Syntax?
- BoolQuestions: Does Dense Retrieval Understand Boolean Logic in Language?
- LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval
- WikiQA: A Challenge Dataset for Open-Domain Question Answering
- Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks?
- Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Models
- Predicate-Argument Based Bi-Encoder for Paraphrase Identification
- PAWS: Paraphrase Adversaries from Word Scrambling
- Drop your Decoder: Pre-training with Bag-of-Word Prediction for Dense Passage Retrieval