Questions like "Where was the director of Inception born?" need two lookups. First the system must find who directed the film, then find that person's birthplace. The second lookup depends on a name, the "bridge entity", that the question never mentions. Recent work shows a language model often already knows this intermediate answer partway through its layers. A small network can map those middle-layer states straight into a dense retriever's vector space, a system that finds documents by nearness of learned embeddings. This fetches the second-hop document without ever writing out a search query, and it runs almost as fast as single-step retrieval. The weak point is that a dense vector is a blurry target. It can land near documents about similar topics without landing on the exact person or place the chain requires. It also cannot bridge phrasings that the embedding space places far apart.
We propose to read two things from the same hidden state instead of one: a dense vector, and a small, weighted set of vocabulary terms. The terms split into "invariant" words, meaning entity names that must match exactly, and "variant" words, meaning concept terms that may appear under synonyms. Retrieval then combines exact keyword matching on the invariant terms with dense matching on the rest. The whole next-hop query comes out of one forward pass, with no text generation. Because entity names are the part of multi-hop queries that dense retrievers blur, we expect the gains to concentrate on questions where the question and the second-hop evidence share few words. We also run the method on both a causal decoder and an encoder-decoder. This tells us whether "middle layers carry next-hop intent" is a general property of transformers or an artifact of left-to-right models.
Core Claim
The claim has two parts. First, a lexical readout of the bridge entity from middle-layer states, fused with the dense readout, will beat dense-only latent retrieval on second-hop recall. The gain should be largest when question and gold evidence overlap little, and the method should keep single-pass latency. Second, the layer at which this readout works best differs between a causal decoder and an encoder-decoder's bidirectional encoder, so the choice of layer must be re-derived for each architecture rather than copied.
The method comes from three sources, each covering a gap in the others:
- Optimizing Multi-Hop Document Retrieval Through Intermediate Representations (L-RAG) supplies the core mechanism. It identifies extraction, processing and refinement stages across layers, and trains a multilayer perceptron from middle-layer states into the query space of Contriever, a dense retriever. It has no handling of term variation or exact entity matching, and it was studied only on causal decoders.
- Lexical Diversity-aware Relevance Assessment for Retrieval-Augmented Generation (DRAG) supplies the invariant/variant/supplementary split of query components. DRAG applies it in text, through a separate small model, and uses it to recalibrate generation at a single retrieval stage. It does not use the split to formulate queries across hops. We move the split into the retrieval head itself.
- DRAGIN motivates the lexical readout from the opposite side. It builds next-hop queries from the tokens with the highest self-attention weights in the existing context. It therefore cannot emit a bridge entity that has not yet appeared in text. Our head scores the full vocabulary, so it can.
Research Program
We attach two heads to a chosen layer $\ell$ of a frozen model.
The dense head follows L-RAG. It maps the pooled hidden state $h_\ell$ to a Contriever query vector.
The lexical head maps $h_\ell$ to non-negative weights over the tokenizer vocabulary, keeping the top few dozen. Each kept term also gets a binary "invariant" flag.
The second hop is scored as
$$s(d) = \alpha \, \langle f(h_\ell), E(d) \rangle + (1-\alpha) \Big( \sum_{t \in I} w_t \, \mathrm{BM25}(t,d) + \lambda \sum_{t \in V} w_t \, \mathrm{BM25}(t,d) \Big)$$
The symbols are as follows:
- $d$ is a candidate document and $E(d)$ is its Contriever embedding.
- $f$ is the dense head.
- $I$ and $V$ are the invariant and variant term sets from the lexical head, and $w_t$ is the head's weight for term $t$.
- BM25 is standard keyword matching.
- $\alpha$ and $\lambda$ are mixing weights tuned on development data. Invariant terms carry full weight because an entity name must match literally.
Supervision comes from existing annotations. In 2WikiMultiHopQA (Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps), Wikidata triples name the bridge entity, so its tokens are the invariant targets. In HotpotQA (HotpotQA), the title of the second gold paragraph is the invariant target. For both datasets, the salient non-entity words of the gold second-hop sentence are the variant targets. The dense head is trained contrastively against the gold second-hop document, as in L-RAG.
Unlike DRAG, we need no GPT-4-labelled decomposition data, because the invariant/variant distinction comes from the dataset structure.
The architectural study runs the same probes on Llama-3-8B-Instruct and on Flan-T5-XL, the 3B encoder-decoder that IRCoT showed can outperform a 175B model when retrieval is iterative. We sweep $\ell$ over all layers of each. For the decoder we also record the fraction of attention on the first token at each layer. That fraction measures the "attention sink" artifact that From Attention to Activation attributes to causal masking. We then test whether heavily sink-dominated layers make worse readout sites. For the encoder we check whether the best readout layer sits in the upper layers. There, Topics in Contextualised Attention Embeddings found BERT's attention forming coherent topic clusters. That work concerns topics rather than entities, so it motivates the hypothesis but does not predict the answer.
Baselines
Naive reference. One-shot retrieval uses the original question, with BM25, Contriever, and their score fusion, followed by the same reader. This measures how much any second-hop mechanism adds.
L-RAG. This is the strongest direct competitor, because it shares our cost profile of one forward pass and no generated queries. The file reports its MuSiQue accuracy as 0.352 and its latency on 2WikiMultiHopQA as about 2.8 seconds per query, against more than 20 seconds for IterRetGen. No reported result was found on Lacuna for L-RAG on HotpotQA. We re-implement it on the same frozen models, the same layer sweep, the same training splits and an equal head-training budget. Beating it means higher second-hop Recall@5 and answer F1 on MuSiQue and 2WikiMultiHopQA at statistically indistinguishable latency.
IRCoT. This is the accuracy ceiling for text-based iteration. The file reports it improving retrieval recall by 11–21 points and answer F1 by up to 15 points over one-step retrieval. We run it with the same reader and corpus. We do not expect to beat it on accuracy everywhere. Success means closing most of the gap between L-RAG and IRCoT at a fraction of IRCoT's cost.
DRAGIN. This is the strongest method that also uses internal signals. The file reports an F1 of 0.3931 on 2WikiMultiHopQA with LLaMA2-13B. We run DRAGIN on Llama-3-8B for a matched comparison and report the published figure only as context.
Ablations. Three ablations isolate the key ingredient. The first is a dense-only head, which is L-RAG. The second is a lexical-only head. The third is the hybrid with the invariant flag removed, so all terms are weighted equally. If the fused head without the invariant/variant split performs as well as the full head, the DRAG-derived decomposition is not pulling its weight.
Why not just…?
Why not just fuse L-RAG's dense retrieval with BM25 on the original question? This is the cheapest hybrid and a genuine threat on HotpotQA, where many questions leak the bridge entity. Keyword matching on the question, however, can only match words that are already in the question. By construction, the second-hop bridge entity is absent. We include this fusion as a baseline. We predict it matches our method in the high-overlap stratum and loses in the low-overlap stratum. If it does not lose there, the lexical readout adds nothing.
Why not just let the model generate the bridge entity in a few greedy tokens and search on that? This is the most serious alternative, because a handful of decoded tokens is cheap. It is also a stripped-down form of what IRCoT does. We run it with the same reader and a matched latency budget, and we report the result either way. Our method can still win for two reasons. Decoding commits to a single string, while the head keeps a weighted set of alias candidates. Decoding also adds sequential passes that compound over three- and four-hop MuSiQue chains. If short decoding matches us, the honest conclusion is that the contribution is the architectural analysis rather than the head.
Why not just use a static query rewriter such as RaFe? RaFe trains small rewriters from reranker preferences and reports 2–3% answer accuracy gains. It rewrites only the initial query, though, and the next-hop terms depend on evidence it has not yet seen. We run RaFe-style rewriting in front of one-shot retrieval as a baseline. We expect its gains to be small on MuSiQue, which MuSiQue built specifically to defeat single-hop shortcuts.
Why not just use a bigger model? MultiHop-RAG reports that GPT-4 reaches only 0.56 accuracy with retrieved chunks, against 0.89 with gold evidence. Retrieval, not reader scale, is the bottleneck, so we hold the reader fixed and vary only retrieval.
Proposed Experiments
We evaluate on HotpotQA (full-wiki setting), 2WikiMultiHopQA and MuSiQue-Ans. We train the heads on HotpotQA and 2WikiMultiHopQA and test MuSiQue zero-shot for transfer. The metrics are:
- second-hop Recall@5 and full-chain recall;
- answer exact match and F1 with a fixed reader;
- wall-clock latency per query on identical hardware.
Lexical-gap stratification. We bucket test questions by the content-word overlap between the question and the gold second-hop document. The core claim predicts that our method's margin over L-RAG grows as overlap falls. A flat margin across buckets would falsify the lexical-gap explanation.
Two controlled stress tests. Both use counterfactual corpora; this construction is ours.
- In the variant stress test, we paraphrase the non-entity words of gold second-hop documents while keeping entity names intact. Dense heads should degrade, and the variant channel should degrade gracefully.
- In the invariant stress test, we replace entity mentions with Wikidata aliases from 2WikiMultiHopQA. This checks whether the lexical head's weighted term set covers aliases or overfits to canonical titles.
Architectural check. We plot readout recall against layer depth for Llama-3-8B and for Flan-T5-XL's encoder, alongside the decoder's per-layer first-token attention fraction. The second claim holds if the best readout layers differ in relative depth between the two architectures, or if the encoder gives higher peak recall at matched scale. It fails if the curves coincide, in which case L-RAG's layer finding transfers as-is.
Decisive result. The decisive result is a head-to-head against the few-token bridge-entity decoding baseline on three- and four-hop MuSiQue questions at matched latency. That is the regime where committing to one generated string at each hop should hurt most, and it is where this proposal either earns its place or does not.
References
- Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
- Lexical Diversity-aware Relevance Assessment for Retrieval-Augmented Generation
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models
- Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- From Attention to Activation: Unraveling the Enigmas of Large Language Models
- Topics in Contextualised Attention Embeddings
- RaFe: Ranking Feedback Improves Query Rewriting for RAG
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries