Training Dense Retrievers on What BM25 Misses

Hybrid search engines combine two kinds of scorer. One counts exact keyword overlap; BM25 is the standard version, and it gives extra weight to rare words. The other compares learned vector embeddings that capture meaning, the "dense" retriever. The usual recipe trains the two separately and then adds their scores with a hand-tuned weight. This note argues that the recipe is quietly broken in specialized domains such as biomedicine. When the dense encoder has been pre-trained on domain text, it already learns much of what BM25 provides, such as sensitivity to rare gene names, drug codes, and version strings. Adding BM25 on top then counts the same evidence twice. This is a retrieval version of the "double correction" error that the fairness and long-tail literature has documented when two independent fixes for the same skew are stacked.

We propose to fold the BM25 score into the dense retriever's training objective as a fixed offset. The dense model then learns only the relevance signal BM25 does not already supply, and the same combination is used at test time. The motivation is practical. In biomedical search, pure dense retrievers lose badly to BM25, and post-hoc mixing is the main way practitioners recover. Clinical question answering is limited by what the first-stage retriever surfaces. If double counting is real, then domain pre-training and hybrid fusion currently partly cancel each other, and fixing that should raise both retrieval quality and downstream exam accuracy.

Core Claim

The overcorrection lens comes from Balanced Meta-Softmax for Long-Tailed Visual Recognition. That paper shows a classifier's softmax encodes the training prior. Correcting the prior with both resampling and a reweighted loss penalizes frequent classes twice. The authors' fix is to put the known prior inside the softmax as a logit offset, so the learned part models only the remainder. Towards Understanding Gender Bias in Relation Extraction found the same pattern empirically: stacking data equalization with embedding debiasing hurt both accuracy and group disparity relative to either intervention alone. We do not reuse their demographic setting. We borrow the structural lesson: two corrections aimed at the same deficit interfere unless one is trained conditioned on the other.

The retrieval analogue is direct. Let $s_{\text{bm25}}(q,d)$ be the BM25 score of document $d$ for query $q$, and let $s_\theta(q,d)$ be the dense encoder's inner-product score with parameters $\theta$. We train with a contrastive softmax over a positive document and negatives $d'$:

$$p_\theta(d \mid q) = \frac{\exp\big(\lambda\, s_{\text{bm25}}(q,d) + s_\theta(q,d)\big)}{\sum_{d'} \exp\big(\lambda\, s_{\text{bm25}}(q,d') + s_\theta(q,d')\big)}$$

Here $\lambda$ is a fixed weight on the lexical prior. At test time the hybrid score is exactly $\lambda s_{\text{bm25}} + s_\theta$.

Gradients reach $\theta$ only for relevance BM25 fails to explain. For example, where BM25 already ranks the positive first, the dense encoder receives little pressure to relearn exact matching. The hybrid then becomes jointly optimized, rather than two independently trained scorers glued together by grid search. Sparse, Dense, and Attentional Representations for Text Retrieval explicitly leaves this joint optimization open.

The testable claims are three. First, redundancy between BM25 and the dense scorer is higher for domain-pretrained encoders than for general ones. Second, post-hoc interpolation therefore gains less from domain pre-training than it should. Third, residual training recovers that lost gain.

Research Program

The first part is diagnostic. We will initialize the same dual encoder either from a general BERT checkpoint or from a biomedically pre-trained one. BioBERT is used in What Disease does this Patient Have?. Both are trained on identical data: the synthetic PubMed query–abstract pairs of Hybrid First-stage Retrieval Models for Biomedical Literature, or a subsample sized to our compute. For each encoder we measure redundancy with BM25 in two ways: the per-query rank correlation of the two scores over candidate pools, and the overlap of their top-100 lists. We also bucket test queries by whether they hinge on a rare alphanumeric entity. Ma et al.'s "JASPAR 2016" versus "JASPAR 2010" is the canonical example. The hypothesis is that domain pre-training raises redundancy specifically in the rare-entity bucket.

The second part is the method itself, residual hybrid training. We will vary $\lambda$ over a small grid. We will also test one variant in which $\lambda$ is learned jointly, to check whether the fixed-prior design matters. Hard negatives are mined from the current hybrid score rather than from BM25 alone. BM25-mined negatives would otherwise teach the dense model to fight BM25's specific mistakes, which is a second, hidden correction stacked on the first.

The third part studies overcorrection in the opposite direction, which the long-tail literature warns about. If $\lambda$ is large, the dense side may learn to ignore lexical cues entirely. The system then has no fallback when BM25 itself fails, for instance because of tokenization of chemical names or because of synonymy. We will report results per bucket for queries where BM25 recall@100 is zero. The method must not lose to the plain dense encoder there.

The fourth part addresses the lexical annotation bias that Ma et al. flag in BioASQ: relevance labels were produced with keyword search, so they favor lexical systems. To get an evaluation that this bias does not distort, we feed each retriever's top passages from medical textbooks into a fixed reader on MedQA. We then judge retrieval by downstream answer accuracy rather than by keyword-curated labels.

Baselines

The naive reference points are BM25 alone and the dense encoder alone, each with general and domain-pretrained initialization. On BioASQ 8 Batch 2, Ma et al. report MAP of 0.2718 for BM25 against 0.1173 for their dense encoder. That is the gap any hybrid must close.

The strongest existing first-stage competitor is post-hoc hybrid fusion. We implement it two ways. One is Ma et al.'s concatenated vector, which casts BM25 as a sparse dot product joined to the dense embedding for single-stage nearest-neighbor search. The other is the linear score interpolation of Luan et al. In both, the interpolation weight is tuned on the same development split, with the same number of trials, as our $\lambda$. Both use the identical encoder checkpoint, training pairs, negatives budget, and GPU-hours. The only difference is whether BM25 is present during training.

The strongest complete system is Ma et al.'s HybridRerank, which adds a BERT cross-attention reranker. It reports an average MAP of 0.4056 across the five BioASQ 8 batches, against 0.4043 for the best official systems. We will place the same reranker on top of our first stage and on top of theirs. Beating HybridRerank's 0.4056 under a matched reranker, and showing higher first-stage recall@100 feeding it, would count as a win.

On MedQA USMLE, the reference is BioBERT-Large reading BM25-retrieved passages, which reaches 36.7% accuracy in Jin et al. We keep that reader fixed and change only the retriever. On MS MARCO, used as a general-domain control, ME-HYBRID reports MRR@10 of 0.343 (Luan et al.). We do not expect to beat it there. We expect residual training to help less in the general domain, where encoders are less redundant with BM25. For BEIR's biomedical subsets (TREC-COVID, NFCorpus, SciFact, BioASQ), no reported result was found on Lacuna for per-dataset nDCG@10, so we will run BM25 and the hybrids ourselves.

The key ablation trains the dense model with BM25 inside the softmax but discards BM25 at test time. If this ablation does well, the dense model has simply learned something different rather than something complementary. The full method should beat both this ablation and post-hoc fusion.

Why not just…?

Why not just tune the interpolation weight carefully per domain? This is the real threat. If a well-tuned $\alpha$ on a domain-pretrained encoder matches residual training, then the double counting is harmless and the proposal fails. The diagnostic experiment settles it. We will compare the best post-hoc hybrid against the residual hybrid under an identical tuning budget, separately for general and domain-pretrained encoders. Our hypothesis predicts that the advantage of residual training grows with domain pre-training and concentrates in the rare-entity bucket. A uniform small difference would refute it. The reason to expect a gap is that post-hoc tuning chooses one scalar after training, whereas residual training changes what the encoder spends its capacity on. Luan et al. show that capacity is a real constraint for single vectors.

Why not just use BM25 plus a cross-encoder reranker? BEIR finds this among the most robust zero-shot setups. A reranker, however, cannot recover documents the first stage never retrieves. Jin et al. attribute much of MedQA's ceiling to exactly that retrieval bottleneck, and Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text attributes 32% of its errors to retrieval noise. We therefore report first-stage recall and also rerank every first stage with the same model, so any gain must survive reranking.

Why not just drop retrieval and rely on a domain-pretrained or larger model? TheoremQA shows that open models stay below 15% on theorem-driven questions, and that supplying definitions in the prompt does not help much. BioBART improves generation but leaves a gap on clinical dialogue. These results do not prove retrieval is necessary for MedQA. Our closed-book reader row tests this directly.

Proposed Experiments

The decisive checks are four. Retrieval experiments run on BioASQ 8 (MAP, recall@100), the BEIR biomedical subsets (nDCG@10), and MS MARCO Passage (MRR@10) as a control. Downstream accuracy is measured on MedQA USMLE with a fixed reader.

First, a two-by-two table crosses initialization (general or domain-pretrained) with fusion (post-hoc or residual). Our hypothesis predicts a positive interaction: residual training should help domain-pretrained encoders more. No interaction would falsify the double-counting story.

Second, per-bucket results should show that the gain comes from rare-entity queries. There must also be no loss on queries where BM25 retrieves nothing, which is the overcorrection check.

Third, the MedQA reader's accuracy should rise with residual retrieval even where BioASQ MAP gains are small. That would indicate the label-free evaluation sees improvements that lexically curated labels hide.

Fourth, a sweep over $\lambda$ should trace an inverted U. A small $\lambda$ reproduces independent training; a large $\lambda$ starves the dense side. Locating the peak relative to the post-hoc optimal $\alpha$ tells us how much double counting the standard recipe was doing.