Agreement-Gated Hybrid Retrieval for Multi-Hop Search over Cold Documents

Multi-hop retrieval answers a question by fetching a chain of documents, where each document supplies the clue needed to find the next one. Each step can go wrong, and a mistake at an early step pulls in distractor documents that the later steps then build on. Two kinds of retriever are available at every step. Keyword matching, such as BM25, is exact on names but blind to paraphrase. Dense embedding search, which maps text to vectors and matches by meaning, handles paraphrase but blurs rare names together. Current multi-hop systems pick one of these and use it at every step, or they mix the two scores with a single fixed weight. This note proposes choosing the mix separately at every step of every query, using one cheap and observable signal: whether the keyword retriever and the embedding retriever agree on what they found. When they disagree, the system also narrows how many candidates it carries forward. We expect this to matter most for cold documents, meaning documents added after training whose entities neither the embedding model nor any precomputed knowledge graph has seen.

The question is practical. The strongest multi-hop systems either rely entirely on learned embeddings, as GRITHopper and MoPo do, or depend on knowledge-graph triplets extracted ahead of time, as HELP, QSGNN, and HLG do. Both designs degrade when new documents arrive. Embedding models are entity-blind on unseen names, and graph methods have no edges for new documents until extraction is rerun. A keyword index absorbs a new document instantly. If we knew when to trust it inside a reasoning chain, we would get most of the benefit of hybrid retrieval with no re-indexing.

Core Claim

The best balance between keyword and embedding retrieval is not a property of the corpus, and it is not mainly a property of hop depth. It is a property of each (query, hop) pair. A few signals available at retrieval time can predict it. The central signal is how much the top-k lists of the two retrievers overlap. Other signals are the rarity (inverse document frequency) of the rarest term in the current hop query, and whether that query contains an entity string copied from the previously retrieved passage. We also claim that the same disagreement signal is a good trigger for pruning: noise accumulates mostly at hops where the two retrievers disagree, so carrying fewer candidates forward at exactly those hops should cut distractors without cutting recall.

The research file states a hypothesis that keyword matching helps at the first hop while embeddings help at deeper "conceptual" hops. We treat this as an open question rather than an assumption. There is a plausible reason it could run the other way. In chain retrieval, the query for the second hop usually contains a bridge entity name copied verbatim from the first-hop passage, which is exactly where keyword matching is strongest. Measuring which way the optimal weight moves with depth is itself a first result.

Mechanism

At hop $k$, a candidate document $d$ gets the score

$$s_k(d) = \alpha_k\,\hat s_{\text{sparse}}(q_k, d) + (1-\alpha_k)\,\hat s_{\text{dense}}(q_k, d),$$

Here $q_k$ is the hop query: the original question joined with the passages retrieved so far, as in the MDR-style loop used by MoPo. The hatted terms are keyword and embedding scores normalized within the candidate pool. The weight $\alpha_k \in [0,1]$ is predicted per query and per hop by a small model, either logistic regression or a shallow tree, with a handful of input features. The first feature is the Jaccard overlap of the two retrievers' top-20 lists. The others are the maximum inverse document frequency among the terms of $q_k$, an indicator for a copied entity span, the hop index, and the score gap between the first and tenth result in each list.

We train this small model on development questions with an oracle target. For each (query, hop), the target is the $\alpha$ on a grid that gives the gold next document its best rank. Because the model has only a few dozen parameters, it should transfer to new domains and cold documents far better than a retriever fine-tuned end to end. The beam width $b_k$, meaning the number of partial chains kept, shrinks linearly with disagreement. This applies the query-guided pruning idea of QSGNN to a signal that needs no graph and no training on graph triples.

Relation to Prior Work

Prior work on hybrid retrieval supplies the method and also sets the limits we start from. Sparse, Dense, and Attentional Representations for Text Retrieval shows that fixed-length embeddings lose their margin between relevant and irrelevant documents on long passages, and that a static BM25 plus multi-vector interpolation recovers it. Hybrid First-stage Retrieval Models for Biomedical Literature shows that keyword scores can be folded into a single vector search, and that pure embedding retrieval collapses on specialist names. Both use one global weight on single-hop tasks. We vary the weight per hop and per query, and we test inside reasoning chains.

SymRAG is the closest work on adaptivity. It routes whole queries to symbolic, neural, or hybrid paths before retrieval begins. We make the decision inside the chain, and our routing signal is retriever agreement rather than a query-complexity score.

HELP already combines graph consensus scoring with dense passage retrieval and controls noise with beam search. We drop the triplet index, which is what makes HELP fail on cold documents. The file notes that HELP's seed selection depends on lexical overlap, so cold documents are a direct stress test for it.

The cold-start literature motivates the evaluation regime. ColdRAG reports that LLM reranking cannot recover candidates that the first-stage retriever missed. KALM4Rec uses only keyword-weighted, one-hop graphs. Both point to the first-stage hybrid retriever as the bottleneck we target. We do not claim to improve LLM reranking.

BEIR constrains the design. It shows that BM25 stays competitive out of domain, so any learned component must stay small enough not to overfit the domain it was trained on.

Research Program

We build one shared multi-hop retrieval loop with three interchangeable components: a BM25 index; an embedding retriever, MDR-style and fine-tuned on the training split; and the fusion and beam controller. This setup isolates the scoring rule from everything else.

We then construct a cold-document protocol on HotpotQA and MuSiQue. We select a set of entities, remove every document mentioning them from the embedding retriever's fine-tuning data and from any graph-construction pass, and insert those documents into the index only at test time. Test questions whose gold chain passes through such a document form the cold subset. The embedding model encodes cold documents, but it has never trained on their names. Graph methods see these documents only as passages, unless we pay for re-extraction, and in that case we report the extraction cost.

The analysis comes in three parts. First, a headroom study compares four ways of setting the weight: a single global $\alpha$, the best $\alpha$ per hop index, the oracle $\alpha$ per (query, hop), and the learned predictor. The study also reports how the optimal $\alpha$ moves with hop depth, separately for warm and cold subsets. Second, a noise audit measures, at each hop, the fraction of the beam that lies off every gold chain, and relates it to retriever disagreement. Third, a transfer test applies the controller, trained on HotpotQA, to MuSiQue, MultiHop-RAG, and BioASQ-style terminology-heavy queries without retraining.

As an exploratory extension, we may cast the Amazon item cold-start setting used by ColdRAG as two-hop retrieval: from the descriptions of warm items a user interacted with, to the text of cold items. This extension is secondary and is not part of the core claim.

Baselines

The naive reference point is BM25 run inside the same multi-hop loop. It needs no training and absorbs cold documents for free, so it is the honest floor on the cold subset. Next comes the embedding-only version of the same loop, with the MDR-style retriever fine-tuned on the same training split. On HotpotQA, MoPo reports MDR at EM@100 of 79.70, and MoPo itself at 82.20. MoPo is the strongest reported embedding-based retriever on HotpotQA in the file, so we will reproduce it under the same training data. Beating it on the cold subset while staying within one point of it on warm questions would count as success.

GRITHopper is the strongest decomposition-free multi-hop competitor. It reports Hits@1 of 95.44 on MoreHopQA against MDR's 82.16. We run it on our cold split from its public setup with the same corpus snapshot and compare Hits@1 per hop. For MuSiQue, the file contains no absolute retrieval number for HELP, QSGNN, or GRITHopper; QSGNN reports only a relative 33.8% Recall@5 gain on 4-hop questions. We therefore produce our own numbers for these methods under a shared evaluation.

On MultiHop-RAG, the benchmark paper reports Hits@10 of 0.7467 and Hits@4 of 0.6625 for its best reranked retrieval (MultiHop-RAG). We treat these figures as the bar to clear without a cross-encoder reranker.

HELP is the strongest graph-plus-dense hybrid. We run it twice on the cold split: once with no triplets for cold documents, and once with triplets re-extracted, reporting both wall-clock and extraction cost. All learned systems get the same tuning budget, measured as a grid size and a number of development questions.

Two ablations isolate the key ingredient. In the first, the learned predictor uses hop index alone, which is the per-hop static weight. In the second, the predictor keeps all features except retriever agreement, and the beam width is held fixed.

Why not just…?

Why not just tune a separate fixed weight for each hop depth? This is the most serious threat, and the proposal is built around it. If the oracle per-(query, hop) weight beats the best per-hop-index weight by less than about one point of recall on the cold subset, then per-hop tuning is the right answer. We would report that as the finding and drop the predictor. The headroom study answers this before any learned component is trusted.

Why not just use reciprocal rank fusion, which needs no weights at all? Reciprocal rank fusion merges the two ranked lists by summing inverse ranks. It is robust, but it is symmetric by construction, so it cannot favor keyword matching on a rare copied entity name. We include it as a baseline. If it matches the learned predictor, that would suggest disagreement matters for pruning but not for weighting, and the beam ablation would show that.

Why not just fine-tune a stronger embedding retriever such as GRITHopper on more data? On warm questions it may well win. The case for our method rests on the cold subset, where BEIR and the BioASQ collapse of pure embedding retrieval in Hybrid First-stage Retrieval Models (MAP 0.1173 against BM25's 0.2718) suggest that embedding models fail on unseen names. If GRITHopper's cold-subset drop is small, our motivation weakens, and we will say so.

Why not just let an LLM decompose the question and rewrite each sub-query? Rewriting can bridge vocabulary gaps. Its cost is reported directly, though: 1 hour 24 minutes for GPT-4o decomposition on MuSiQue, against 6 minutes 42 seconds for GRITHopper (GRITHopper). Rewriters trained from ranking feedback, as in RaFe, also inherit the biases of the reranker that trained them. We run a decomposition baseline on a 500-question sample to measure accuracy per unit of cost, rather than assuming it loses.

Proposed Experiments

The decisive checks come in four parts. The first is the headroom gap between per-hop-index and per-(query, hop) oracle weights, on warm and cold subsets of HotpotQA and MuSiQue. The second is recall at 2, 3, and 4 hops on MuSiQue, comparing the learned controller against BM25-only, embedding-only, MoPo, GRITHopper, and HELP, with and without re-extraction. The third is the per-hop distractor fraction, testing whether agreement-triggered beam narrowing reduces it without lowering final-chain recall. The fourth is zero-shot transfer to MultiHop-RAG and a terminology-heavy biomedical subset.

The idea is confirmed if three things hold together. The headroom gap must be substantial. The learned controller must recover most of that gap. And the cold-subset advantage over embedding-only and graph baselines must exceed the warm-subset advantage. The idea is refuted if a per-hop static weight, or reciprocal rank fusion with a fixed beam, performs within noise of the full method on cold questions.