Questions like "In what city was the director of Inception born?" defeat ordinary retrieval-augmented generation (RAG, where a model looks up documents before answering). The document about Nolan's birthplace only becomes findable once the model has worked out "Nolan." Text-based fixes such as having the model write a reasoning sentence and use it as the next search query do work, but they are slow. Recent work shows a shortcut. Partway through its layers, a language model's hidden state already encodes what it would need to look up next, so that state can be mapped directly into a retriever's vector space without generating any text. That work picks one fixed middle layer and handles a single extra hop. Separate work on layer selection shows that the most useful layer depends on the input: some prompts are compressed early and others late.
This note asks a simple question. As a model works through hop 1, hop 2 and hop 3 of a question, does the layer where "what to look up next" is most readable stay fixed, or does it move? If it moves predictably, a cheap per-hop rule for choosing the layer should beat both a fixed layer and a learned static blend of layers. That rule would make latent retrieval work on 3- and 4-hop questions while keeping the latency of a single-step RAG call. If it does not move, the measurement itself settles an open question and tells practitioners that a static layer choice is enough.
Core Claim
We claim that next-hop retrievability peaks in a band of layers just before the model "commits" to its output vocabulary. We also claim that the depth of this band varies across queries and shifts as resolved evidence accumulates in context. "Commitment" here means the layer at which a logit-lens reading of the hidden state stops changing much relative to the final layer. A logit-lens reading decodes an intermediate hidden state through the model's output head.
If this is right, a training-free, per-hop selection rule should recover most of the gap between a fixed layer and an oracle that picks the best layer per query. The rule is simple: extract the representation a fixed offset before the commitment depth. An oracle that picks the best layer per query is an upper bound and can only be measured offline, using gold supporting documents.
The new piece is not intermediate-layer retrieval itself, which already exists. It is two things together: a hop-resolved map of where retrievable intent lives, and an input-dependent layer choice inside an iterated latent-retrieval loop.
Relation to Prior Work
The method we build on is L-RAG. L-RAG describes an extraction, processing and refinement pattern across depth, then trains a small multi-layer perceptron (a projector) that maps one fixed middle-layer hidden state into the Contriever embedding space. It retrieves the second-hop document from a single forward pass. We keep its projector idea but change three things. We train the projector across all candidate layers, choose the layer per query and per hop, and iterate beyond two hops.
Two papers on layer utility motivate the input-dependence hypothesis:
- Does Representation Matter? shows that the middle layers of Transformers compress inputs and that this compression is bimodal across prompts in specialized domains. A single fixed layer is therefore likely wrong for a sizable fraction of queries.
- Uncovering the Latent Potential of Deep Intermediate Representations selects layers offline and globally using ridge-regression fit and isotropy. Its LOES method serves as our static, data-driven layer-choice competitor. The same paper constrains us by warning that heavily distilled models leave little intermediate-layer headroom.
LFD supplies the logit-lens machinery we reuse for the commitment signal. LFD uses an internal knowledge score to find the layer that leans most on retrieved context, then blends that layer into decoding. It acts only after retrieval and is not hop-aware; we use the same family of signals to decide where to retrieve from.
DRAGIN also reads internal states, using attention and entropy to decide when and what to retrieve. However, it turns attention back into surface keywords and relies on final-layer uncertainty. We retrieve with continuous vectors and use an intermediate-layer signal instead.
T2MLR suggests that multi-step reasoning recurs through middle layers. That supports, but does not establish, the possibility that each hop is resolved at a different depth.
Research Program
The system is an iterated latent retrieval loop. At hop $k$, the model receives the question plus the documents retrieved so far and runs one prefill pass with no generation. A projector $P(h_\ell, \ell)$, conditioned on a layer-index embedding, maps the last-token hidden state $h_\ell$ at the chosen layer $\ell$ into Contriever space and retrieves the top document. That document is appended, and the loop repeats until a stopping rule fires. The model then generates the answer once. A single shared projector trained on all layers keeps parameter cost close to L-RAG's. Supervision comes from gold next-hop documents in the training splits of MuSiQue and 2WikiMultiHopQA, and every layer receives training signal so that the layer choice can be studied separately from projector quality.
The first deliverable is a measurement study, the hop-depth map. On the MuSiQue development set, which composes 2- to 4-hop questions with annotated bridge documents, we compute next-hop retrieval recall for every layer and every hop index. We do this with gold prior hops in context, which separates the layer question from error cascades, and with predicted prior hops. The map tests three things: whether the best layer drifts with hop index, whether it varies within a hop index across queries, and whether it tracks the logit-lens commitment depth.
The second deliverable is the selection rule. For layer $\ell$, define commitment as
$$c_\ell = \mathrm{KL}\big(p_L \,\|\, p_\ell\big),$$
where $p_\ell$ is the logit-lens next-token distribution at layer $\ell$, $p_L$ is the final-layer distribution, and $L$ is the number of layers. The commitment depth $\ell^\dagger$ is the first layer at which $c_\ell$ drops below a threshold $\tau$. We retrieve from $\ell^\dagger - \delta$, with $\tau$ and $\delta$ tuned once on a held-out development split. We also compare two other per-query signals: the prompt-entropy drop of Does Representation Matter? and a per-query isotropy score adapted from LOES. As a secondary ablation, motivated by Attention Values Encode Sentence Semantics Better Than Hidden States, we replace the hidden state at the chosen layer with that layer's attention-value outputs. This tells us whether the substrate matters as much as the depth.
Stopping uses the retrieval score margin and a rise in commitment at shallow depth, which indicates the model already "knows" the answer. We report against an oracle stop that uses the gold hop count to isolate the effect of the stopping rule.
Baselines
The naive reference is one-shot dense RAG. It embeds the question with Contriever, retrieves top-$k$ documents and answers. It shows how large the multi-hop gap is for our backbone. A second naive point is final-layer extraction through the same trained projector, which tests the over-specialization argument directly on retrieval.
The closest existing method is L-RAG with its fixed middle layer. Its paper reports an accuracy of 0.352 on MuSiQue, the highest among the RAG methods it evaluated, and an average latency of about 2.8 seconds per query on 2WikiMultiHopQA, compared with more than 20 seconds for IterRetGen. Because L-RAG is formulated for two hops, we run it two ways: as published, and as the "iterated fixed-layer" variant that loops the same layer for every hop. The iterated variant is the strongest competitor to our specific claim. It shares our loop, projector, training data, backbone, retriever and corpus, and the only difference is the layer choice. Its fixed layer is chosen by best development-set recall, giving it the same tuning budget as our $\tau$ and $\delta$. Beating it means higher per-hop recall on 3- and 4-hop MuSiQue questions and higher answer F1, with significance assessed by paired bootstrap over questions, at matched latency.
The strongest static layer competitor is a learned scalar mix: a softmax-weighted sum of all layers fed to the same projector and trained jointly. LOES-selected layer subsets serve as its training-free counterpart. Both of these baselines isolate the key ingredient, input-dependence. If our per-hop rule does not beat the learned mix, the dynamic idea fails even if both beat the fixed layer.
Text-based iteration with IRCoT and ITER-RETGEN, along with DRAGIN, are run on the same backbone and corpus as accuracy-oriented references, with latency reported alongside. DRAGIN reports an F1 of 0.3931 on 2WikiMultiHopQA with LLaMA2-13B, compared with 0.3076 for FLARE. We will reproduce DRAGIN on our backbone rather than compare against that cross-setup number. For IRCoT, the file gives only relative gains (11 to 21 recall points over one-step retrieval), so no comparable absolute figure is available. The oracle per-query layer is reported as a ceiling.
Why not just…?
Why not just learn a weighted mix of all layers once and be done? This is the real threat. A static scalar mix is cheap and could absorb most of the benefit if the best layer varies only modestly. The hop-depth map answers it directly. If within-hop variance of the best layer is small and there is no drift across hops, a static mix is the right design, and we will say so. The dynamic rule earns its place only if it beats the learned mix on 3- and 4-hop questions, where we expect the drift to matter most.
Why not just loop L-RAG at its fixed layer? That baseline is in the proposal for exactly this reason. The evidence against it is indirect: the bimodal compression in Does Representation Matter? suggests one layer misfits many inputs. Appended documents also lengthen and change the context, which could move the commitment depth. Whether this happens is an empirical question that the gold-context version of the map isolates.
Why not just use IRCoT-style text queries, which are readable and accurate? They may well remain more accurate on the hardest questions. Their cost, though, is one generation step per hop, measured at over 20 seconds per query for IterRetGen on 2WikiMultiHopQA in L-RAG. Hallucinated intermediate text can also derail later retrieval (Wang et al. 2025). We report an accuracy–latency frontier rather than claiming a dominance we have no grounds to expect.
Why not just train a router network to pick the layer? A trained router is a reasonable variant, and we include a small one as an ablation. We lead with the training-free commitment rule because it transfers across checkpoints without new labels. If the router clearly wins, that becomes the method and the commitment rule becomes the interpretable explanation of what the router learned.
Proposed Experiments
The evaluation uses MuSiQue-Ans, with results broken down by 2-, 3- and 4-hop questions, along with 2WikiMultiHopQA and HotpotQA in the full-wiki setting. Metrics are per-hop Recall@5 against gold supporting documents, answer EM and F1, and wall-clock latency per query on fixed hardware. We use two open backbones of different sizes, plus one heavily instruction-distilled model to test the headroom caveat from Uncovering the Latent Potential.
The study has four decisive checks:
- Does the best layer depend on hop and query at all? In the gold-context map, the best layer must shift by more than a couple of layers across hop indices, or vary substantially within a hop index. If neither holds, the dynamic hypothesis is rejected and the paper reports the static result.
- Does the commitment signal predict the best layer? The rank correlation between $\ell^\dagger$ and the oracle best layer per query must be clearly positive on held-out data.
- Does the rule beat the static alternatives where it should? It must beat both the iterated fixed-layer L-RAG and the learned scalar mix on 3- and 4-hop MuSiQue recall and F1. The gain should be concentrated on deeper hops.
- Does the loop stay cheap? Latency must remain within a small multiple of one-shot RAG, set by one prefill pass per hop, and far below text-iteration methods.
Two ablations complete the picture. The attention-value substrate swap tests whether the representation type matters as much as the depth. The oracle-versus-predicted stopping comparison separates hop-count errors from layer errors.
References
- Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
- Unsupervised Dense Information Retrieval with Contrastive Learning
- Does Representation Matter? Exploring Intermediate Layers in Large Language Models
- Uncovering the Latent Potential of Deep Intermediate Representations
- LFD: Layer Fused Decoding to Exploit External Knowledge in Retrieval-Augmented Generation
- DRAGIN: Dynamic Retrieval Augmented Generation based on the Real-time Information Needs of Large Language Models
- T2MLR: Transformer with Temporal Middle-Layer Recurrence
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering