Erasing What the Dense Retriever Actually Uses: Causal Lexical Subspaces for Hybrid Search

Hybrid search engines run two retrievers side by side and add up their scores. One is a keyword matcher such as BM25, which scores a passage by how many query words it contains, weighted by how rare those words are. The other is a dense retriever, a neural network that maps queries and passages to vectors and scores them by inner product. The point of combining them is that they should fail on different queries. In practice they often fail on the same ones, because dense retrievers learn to do a soft version of keyword matching themselves. We propose to find, inside an off-the-shelf dense retriever, the small set of directions it actually relies on when it rewards exact word overlap. We would locate these directions by intervening on the model rather than by training classifiers on it. We would then project them out of the dense vectors at search time, with no retraining. This should leave the dense channel to supply what BM25 cannot, so that the sum of the two scores improves.

The interpretability literature explains why the locating step must be causal. Information that a classifier can read out of a representation is often information the model never uses. Elazar et al. found that probe accuracy on BERT has essentially no correlation with downstream importance. Inside retrieval specifically, MultiContrievers found that gender is easy to read out of Contriever's passage vectors, yet erasing the readable gender subspace did not remove biased rankings. If a lexical-overlap subspace is located the same way, by probing, it will likely be the wrong one. The bet of this proposal is that a subspace located by intervention behaves differently from one located by probing, and that the difference shows up as measurable gains in hybrid retrieval.

Core Claim

Dense bi-encoders contain a low-dimensional linear subspace that causally carries their reward for exact query–passage term overlap. Removing this subspace from the query vector at inference makes the dense retriever more complementary to BM25. The hybrid then beats the same dense model fused with BM25 under tuned score interpolation, especially on hard queries and on queries with vocabulary mismatch. The same gain should not appear when the removed subspace is found by a conventional probe, or when it is a random subspace of equal size. The low-dimensionality expectation comes from Hernandez and Andreas, who found linguistic features in BERT concentrated in subspaces of fewer than about 10 rotated dimensions, and from Hao and Linzen, whose single "number" direction flipped verb agreement when reflected.

The claim matters because the bi-encoder score is an inner product. Write the query vector as $q$, the passage vector as $d$, and an orthogonal projector onto the found subspace as $P$. The edited score is then $q^\top (I-P) d$. The edit costs nothing at search time and needs no re-indexing, because only the query side changes. It also turns a diagnostic finding into a deployable change.

Relation to Prior Work

Two papers motivate the gap. On Complementarity Objectives for Hybrid Retrieval named the "Intersection Trap": dense models trained alongside BM25 come to imitate it, so the two retrievers return overlapping candidates. Their fix is retraining with an orthogonality loss against sparse vectors plus masking of exact-match tokens. They never examine what their retrained model still encodes or uses. We aim for the same effect post hoc on frozen models, and our analysis can also test whether their training really removed lexical reliance. MultiContrievers is the closest methodological precedent: it applies probing and nullspace erasure to dense retrievers. However, it studies demographic attributes rather than the features that drive relevance, and it never touches hybrids. Its negative result is the main constraint on our design, since it shows that probe-located erasure can do nothing.

Two other papers already open up dense retrievers with sparse autoencoders, which re-express a dense vector as a weighted combination of a few active "concept" units. Decoding Dense Embeddings builds an inverted index over these concepts. Interpret and Control Dense Retrieval shows qualitative steering by amplifying chosen concepts. Neither measures causal necessity, and neither combines with BM25. Our edit is a measured, selectivity-controlled intervention whose purpose is fusion, not interpretation.

The method we reuse comes from the causal probing line. CausalGym showed that Distributed Alignment Search outperforms linear probes and PCA for finding subspaces that actually change model behavior. Distributed Alignment Search learns a rotation of the representation such that swapping a few rotated coordinates between two inputs reproduces a counterfactual output. CausalGym also stresses control tasks that catch over-expressive interventions. AlterRep showed that probe accuracy can stay above 90% across layers while interventions only work in a few. That pattern is what we expect to see between probe-found and intervention-found lexical subspaces.

Research Program

The first component is counterfactual data. From MS MARCO and Natural Questions training pairs, we build passage variants $d'$ in which the terms shared with the query are replaced by contextually valid synonyms or paraphrases. The variant keeps the meaning and removes the surface overlap. We also build query-side variants $q'$ the same way. Each candidate pair is filtered with a cross-encoder relevance check, so that it still counts as relevant to the original query.

The second component is alignment search. For each retriever, we learn an orthonormal $k$-dimensional subspace $P$, sweeping $k$ from 1 to 32, in the shared embedding space. The training criterion is that swapping the $P$-component of the query between $q$ and $q'$ reproduces the dense score of the paraphrased pair while leaving the complementary component intact. The swap target is the score drop the model itself exhibits under paraphrase, so the subspace explains the model's behavior rather than ground truth. Selectivity is enforced in the CausalGym style. The same search is run on control pairs in which the swapped "variable" is random. A subspace counts only if its interchange accuracy clearly exceeds the control.

The third component is the comparison of erasures and fusion. We compare query-side erasure of three subspaces at matched $k$: the causal subspace, a probe subspace obtained by iterative nullspace projection against a classifier that predicts BM25 overlap from the vectors, and a random subspace. After each erasure we fuse with BM25, retuning the interpolation weights on held-out development data.

Three models anchor the study. The first is Contriever across the 25 MultiContriever seeds, which lets us ask whether the subspace and the gain survive seed variation. MultiContrievers reports up to 6.3 nDCG@10 seed swing on BEIR, so this check is needed. The second is BGE-base. The third is a RoC-style model that we reimplement following On Complementarity Objectives, which serves as a check on whether orthogonality training already removed the causal subspace.

Baselines

The naive reference points are BM25 alone and each dense model alone. BM25 matters because BEIR showed it frequently beats zero-shot dense models across domains. All runs share the same corpus, tokenization and evaluation scripts.

The most important practical baseline is tuned linear score interpolation between BM25 and the unedited dense model. This is what practitioners deploy, and it is hard to beat. Yi Luan et al. report that fusing BM25 with a multi-vector encoder lifts MS MARCO passage MRR@10 to 0.343, from 0.334 for the multi-vector encoder alone. We will tune its weights with exactly the same development set and grid as our edited hybrid. We will also add a per-query adaptive weight variant, so that our gain cannot be explained by a better fusion rule. Beating it means a statistically significant improvement in MRR@10 and Recall@1000 on MS MARCO dev and in nDCG@10 averaged over BEIR, at identical weight-tuning budget.

The strongest existing method is RoC complementarity training. On Complementarity Objectives reports MRR@10 of 38.63 on MS MARCO, against 37.99 for the prior CLEAR plus coCondenser hybrid, and Recall@100 of 90.13% on Natural Questions. Because it retrains the dense model, the fair comparison is twofold. First, our training-free edit applied to a standard dense model should come within the RoC margin over the conventional hybrid at zero training cost. Second, our edit applied on top of a RoC-trained model should add little if RoC already removed the causal subspace, and add measurably if it did not. Either outcome is informative. Matching RoC's 0.64-point MRR@10 gain without retraining would count as success on the first comparison.

The key ablations isolate the causal ingredient. They are erasure of the probe-found subspace and erasure of a random subspace, both at the same dimensionality and fused with the same tuning. If the probe subspace yields the same hybrid gain, the probing-causality distinction is irrelevant here and the claim fails.

Why not just…?

Why not just tune the interpolation weights better? Interpolation rescales the dense channel's opinion; it cannot change which passages that channel ranks highly. If the dense retriever's top candidates largely duplicate BM25's, no weight recovers the passages neither ranks. We will measure top-100 candidate overlap and the recall of the union of the two lists. The proposal predicts that erasure lowers overlap and raises union recall, which reweighting cannot do. If adaptive per-query weighting matches our gains, we will report that this simple alternative suffices.

Why not just retrain with an orthogonality objective, as RoC does? That is a real option and our strongest competitor. It needs full retraining for every model, however, and its geometric constraint against sparse vectors does not show what the dense model has stopped using. Our experiments settle the practical question directly by comparing against reimplemented RoC under matched data. The edit-on-top-of-RoC test also tells us whether RoC's gains come from removing the same mechanism.

Why not just mask high-IDF query terms before dense encoding? This is the training-free cousin of RoC's input masking, and it is a genuine threat because it is trivial to implement. Its weakness is that, at query time, we do not know which terms will overlap with the relevant passage. Masking rare terms also strips topical meaning the dense channel needs, which matters especially for the entity-heavy queries where MMEAD shows BM25 recall on the hardest MS MARCO subset is only 0.6677. We include it as a baseline at matched tuning. If it matches subspace erasure, the simpler method wins and we say so.

Why not just add a cross-encoder reranker? Reranking is complementary rather than competing, since it only reorders whatever first-stage candidates it receives. Better union recall from the first stage should raise any reranker's ceiling. Reranking also costs roughly 314ms per query against about 11ms for a single-stage hybrid in Lin et al.. We will report BM25 plus cross-encoder for context and test whether our first stage improves it.

Proposed Experiments

The first experiment is in-domain. On MS MARCO dev and TREC DL 2019/2020, we report MRR@10, nDCG@10, Recall@1000 and candidate overlap. On Natural Questions, we report Recall@100 against RoC's 90.13%.

The second is out-of-domain, on the BEIR datasets: TREC-COVID, NFCorpus, BioASQ, ArguAna, Robust04 and FEVER, reporting nDCG@10. The file gives no per-dataset BEIR number for tuned BM25-dense interpolation, so we report it ourselves; no reported result found on Lacuna for that configuration.

Two stress sets probe whether the edit damages semantic matching. On the Mismatch Set from Decoding Dense Embeddings, the edited dense model alone should not lose accuracy; no reported numeric result found on Lacuna for dense models on that set. On the MS MARCO Chameleons hardest subset, the hybrid's Recall@1000 should rise above the 0.6677 BM25 reference reported in MMEAD.

Four results decide the claim. First, the causal subspace must pass the selectivity control while staying small, with $k$ under about 16. Second, at matched $k$, causal erasure must improve the hybrid while probe and random erasure do not. That would reproduce the probing-causality gap inside retrieval with a deployable consequence. Third, the effect must hold across most of the 25 Contriever seeds, not just on average. Fourth, the edited dense model alone should lose accuracy mainly on queries with high lexical overlap, and stay flat on mismatch queries, which shows that what was removed is lexical reliance and not general relevance. If probe-found erasure performs as well as causal erasure, or high-IDF masking matches both, we will report the negative result. It would mean that lexical redundancy in dense retrievers is not localized the way syntactic features are in language models.