Language models trained on "A is B" often cannot answer "what is B?", even though the fact is stored in their weights. This failure is called the reversal curse. This note asks whether a similar asymmetry appears in fast neural search engines, and what causes it if it does. The engines in question are dual-encoders: one network turns each passage into a single vector ahead of time, and a second network turns each query into a vector that is matched by inner product. Take a compact retriever specialized for one domain, such as a catalogue, a corporate registry, or a biography collection. Its passages are written about a subject ("Acme's CEO is Dana Reyes") and its training queries mostly ask from the subject's side ("who runs Acme?"). Our hypothesis is that compressing such a passage into one vector lets the subject entity dominate. The object entity ("Dana Reyes") is then only weakly represented, so reverse queries ("which company does Dana Reyes lead?") miss the passage, even though the passage text answers them directly.
If this is true, the usual way of reporting the cost/quality balance of retrievers hides a failure. Retrievers are compared on one quality number against index size and latency, and that number may average over a direction the system serves well and a direction it serves badly. The proposal has two parts. First, it measures directional retrieval accuracy across the efficiency spectrum, from the single-vector dual-encoder through multi-vector models to the full cross-encoder. Second, it separates two possible causes of any gap: the direction of the training queries, which data augmentation can fix, and the capacity of the compressed vector, which augmentation cannot fix.
Core Claim
The reversal-curse literature attributes the asymmetry to the training objective. Next-token prediction fits $P(B \mid A)$ and never $P(A \mid B)$ (An Analysis and Mitigation of the Reversal Curse). The failure disappears when the fact is in the prompt (The Reversal Curse). A dual-encoder has neither problem: its score is a symmetric inner product, and the passage text contains both entities. Any directional gap in a dual-encoder therefore needs a different explanation, and we name two.
The first is training-query direction. If the retriever is trained only on subject-side queries, it may learn that object mentions are irrelevant. The second is a capacity bottleneck. Sparse, Dense, and Attentional Representations for Text Retrieval shows that a fixed-length vector loses its ability to separate the correct passage from wrong ones as passages grow longer. We predict the same loss as the number of facts per passage grows, and we predict it hits the minor entities first.
The central prediction draws on a finding from the reversal-curse literature. The Illusion of Latent Generalization finds that bidirectional training fixes reversal behaviorally by storing forward and reverse facts in separate memory slots, not in one shared representation. If retrievers behave the same way, then reverse-query augmentation should close the gap on short, single-fact passages. It should leave a residual reverse deficit that grows with facts per passage, because the two directions are competing for one vector. Multi-vector and late-interaction models should not show this growth.
The constructive part follows from that prediction. We propose a cheap fix that targets capacity: entity-anchored two-vector indexing. Each passage is stored as two vectors, one pooled over the subject span and one pooled over the object span. This costs twice the single-vector index, compared with the 3x to 8x inflation reported for multi-vector models in Sparse, Dense, and Attentional Representations for Text Retrieval.
Research Program
The test corpora reuse reversal benchmarks from the file and turn them into retrieval tasks. We render each fact as a short passage. We then generate forward queries, which name the subject and ask for the object, and reverse queries, which name the object and ask for the subject. The sources are four:
- the fictitious Name-to-Description biographies, 3,600 fake people, from An Analysis and Mitigation of the Reversal Curse. Because they are fictional, the backbone cannot have memorized them during pretraining.
- the Celebrity Relationship, Author-Work (2,000 pairs) and Company-CEO (1,697 pairs) sets from Rethinking the Reversal Curse of LLMs.
- the 1,573 IMDB parent-child pairs from The Reversal Curse.
For each corpus we build versions with one, two, four and eight facts per passage. This is the controlled variable for the capacity hypothesis.
Domain specialization is the application setting. Following the evidence that compact domain-specialized embedders beat larger general ones (Pre-training Tasks for User Intent Detection and Embedding Retrieval in E-commerce Search), we fine-tune one compact retriever per domain. Training uses forward queries only, matching how domain query logs are typically skewed toward one direction. Separately, we test whether the specialization tax documented for reasoning by Specializing Smaller Language Models towards Multi-Step Reasoning also has a directional form: specializing on forward queries might reduce the reverse accuracy the general model already had.
Cross-domain transfer is the generalization test, not the mechanism. We train reverse-query augmentation on Author-Work only and evaluate reverse retrieval on Company-CEO and the biographies. The question is whether "look at the object entity too" transfers as a domain-general operation, or whether it is learned as a surface pattern specific to one domain. Can Large Language Models Generalize Procedures Across Representations? found the surface-pattern outcome for supervised training in planning tasks.
Baselines
The naive reference points are two.
- BM25, the classical keyword-matching ranker. It is direction-agnostic by construction because it only counts shared words, so it sets the floor for reverse retrieval when no learning is involved.
- A vanilla single-vector dual-encoder fine-tuned on forward queries, which is the condition we expect to show the curse.
The strongest single-vector competitor is the Multi-Margin MSE distilled dual-encoder of In defense of dual-encoders for neural ranking, in which a dual-encoder is trained to match a cross-encoder teacher's score margins. It is the strongest because the teacher reads query and passage jointly and should be direction-agnostic. Distillation may therefore pass symmetry to the student for free. On MS MARCO that paper reports MRR@10 of 0.349 for the distilled dual-encoder, 0.370 for the cross-attention teacher, and 0.376 for distilled ColBERT. We have no reported result on Lacuna for directional or reverse-query retrieval with any of these models.
The strongest efficiency-matched competitors are ME-BERT with learned pooling, at both m=2 and m=8 vectors (reported MS MARCO MRR@10 of 0.334 at m=8 in Sparse, Dense, and Attentional Representations for Text Retrieval), and distilled ColBERT as the late-interaction reference. The cross-encoder serves as the accuracy ceiling.
The key ablations are two.
- Reverse-query augmentation on the vanilla dual-encoder. This tests whether the data alone suffices.
- Learned two-vector pooling (ME-BERT with m=2) against our entity-anchored two-vector pooling. At identical index size, this isolates whether anchoring vectors to entity spans matters.
All models share one backbone, the same training passages, the same total number of training queries (augmented runs replace forward queries rather than adding to them), the same teacher for every distilled variant, and an equal hyperparameter search budget.
Beating the strongest baseline has a specific meaning here. On the eight-fact passages, the entity-anchored model must reach reverse Recall@10 within a small margin of ColBERT while using a two-vector index, and it must clearly exceed the distilled single-vector model and the m=2 ME-BERT model at the same size. It must also do this without losing forward recall or MS MARCO MRR@10 relative to the distilled dual-encoder.
Why not just…?
Why not just add reverse-phrased training queries? This is the real threat. Reverse Training to Nurse the Reversal Curse shows that entity-preserving reversal fixes the generative curse almost completely. The retrieval equivalent is cheap. We include it as a central ablation instead of arguing against it. Our claim survives only if augmentation leaves a reverse deficit that grows with facts per passage, and we expect that growth from the separate-slot storage found in The Illusion of Latent Generalization. If augmentation alone holds reverse recall flat out to eight facts, the capacity hypothesis is falsified. In that case we report the directional audit and the augmentation recipe as the contribution.
Why not just distill from a cross-encoder? The teacher sees both entities jointly, so distillation might transfer symmetry. But distillation mainly improves generalization within the training query distribution (In defense of dual-encoders for neural ranking). If the teacher only ever scores forward queries, it never demonstrates reverse behavior. We test distillation with forward-only and with mixed teacher queries.
Why not just use ColBERT or a hybrid with BM25? Both are likely to fix reverse retrieval. The question is cost. Multi-vector indexes run 3x to 8x larger, and hybrids add a second index and a second scoring path. The proposal earns its place only if a two-vector index recovers most of ColBERT's reverse accuracy. We report the full trade-off curve of reverse accuracy against index size, so a reader can see whether that holds.
Proposed Experiments and Decisive Checks
Metrics are Recall@1, Recall@10 and MRR@10, each reported separately for forward and reverse queries. We also report the directional gap (forward minus reverse), index bytes per passage, and query latency. MS MARCO MRR@10 checks that no method gives up general ranking quality.
The first decisive check is whether a forward-only vanilla dual-encoder shows a reverse gap on the fictitious biographies that BM25 and the cross-encoder do not. If it does not, there is no retrieval reversal curse and the program stops at the audit.
The second check is the slope of the reverse gap against facts per passage, plotted for each architecture with augmentation turned on. We predict a positive slope for single-vector models, near zero for ColBERT and the entity-anchored model, and a positive slope for learned m=2 pooling if anchoring is what matters.
The third check is cross-domain transfer. Augmentation trained on Author-Work should either close the reverse gap on Company-CEO, meaning the retriever learned a general operation, or fail to, meaning it learned a domain-specific pattern and each specialized retriever needs its own reverse data.
The fourth check is the directional specialization tax: the reverse recall of the general backbone measured before and after forward-only domain fine-tuning.
References
- An Analysis and Mitigation of the Reversal Curse
- The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”
- Sparse, Dense, and Attentional Representations for Text Retrieval
- THE ILLUSION OF LATENT GENERALIZATION: BI- DIRECTIONALITY AND THE REVERSAL CURSE
- Rethinking the Reversal Curse of LLMs: a Prescription from Human Knowledge Reversal
- Pre-training Tasks for User Intent Detection and Embedding Retrieval in E-commerce Search
- Specializing Smaller Language Models towards Multi-Step Reasoning
- Can Large Language Models Generalize Procedures Across Representations?
- In defense of dual-encoders for neural ranking
- Reverse Training to Nurse the Reversal Curse