Retrieving the Next Hop Without Words: Testing Whether Hidden-State Retrieval Survives Language Change

Many questions need two lookups: to answer "In what city was the director of Inception born?" a system must first find the director, then find his birthplace. Today's systems handle the second lookup by having the language model write out an intermediate search query in text. Doing this in a language other than English adds failure points the English case never sees. The model may write the query in the wrong language, rely on a translation step, or drift into English in its answer.

A recent English-only result shows the intermediate text can be skipped entirely. The model's middle layers already encode "what to look up next," and a small learned mapping can turn that internal state directly into a search vector. We propose to test whether this shortcut carries over across languages. If the model's internal "next-hop intent" is shared across languages, a mapping trained only on English should retrieve the right second document for an Arabic or Chinese question. That retrieval would need no translation, no language identification, and no generated query. If the shortcut fails, the failure tells us something specific: whether non-English multi-hop failures come from corrupted internal states rather than missing knowledge.

Core Claim

The claim has two parts. First, we claim that a hidden-state-to-retriever projector trained only on English multi-hop data transfers zero-shot to non-English queries. The prediction is that it closes a meaningful part of the cross-lingual multi-hop retrieval gap at a fraction of the latency of text-based iterative retrieval. Second, we claim that where transfer fails, the failure tracks the attention-head misrouting that Sun et al. (2026) identified in closed-book settings. If so, repairing those heads should restore latent retrieval in those languages specifically, without retraining the projector.

The three input directions play different roles:

  1. Multi-hop reasoning gap. This supplies the problem.
  2. Cross-lingual language bias. This supplies the regime where the problem gets worse.
  3. Linguistic tool dependency. This is the constraint. The system may not use machine translation, language identification, or any language-specific query-writing step. Here the "tool" being removed is the intermediate text query itself.

What Exists and What Does Not

The idea sits between two papers that have never been connected. Optimizing Multi-Hop Document Retrieval Through Intermediate Representations (L-RAG) supplies the method. It shows that middle layers resolve next-hop intent before any token is written. It trains a small network (a multilayer perceptron, MLP) to map those states into a dense retriever's embedding space. It reports 0.352 accuracy on MuSiQue and about 2.8 seconds per query on 2WikiMultihopQA, versus more than 20 seconds for iterative generation. It was tested only on English.

Sun et al. (2026) supplies the motivating mechanism. On its XHOPREASON benchmark, factual knowledge lives in language-agnostic neurons. Non-English multi-hop failures come from attention heads that route the query to the wrong intermediate concept. Tuning only the top-40 heads, which they call Mechanistically-Informed Fine-Tuning (MFT), gives a 15% relative gain while keeping MMLU within one point. That work is closed-book; it never touches retrieval.

Our proposal differs from both:

  • Versus L-RAG, we ask whether the projected intent is language-invariant. We also ask whether head misrouting corrupts it.
  • Versus Sun et al., we use their repair as an intervention on a retrieval pipeline. We treat retrieval recall of the bridge document as a far more direct readout of "routing to the right intermediate entity" than final-answer accuracy.

Several other papers set the evidence, the constraint, or the evaluation regime:

  • XRAG provides the evidence that cross-lingual multi-document reasoning, not generation in the target language, is the main bottleneck. It also documents response-language failures. It uses single-round retrieval over supplied documents only.
  • DRAGIN also reads internal states, using token uncertainty and self-attention, to decide when and what to retrieve. It still builds a text query from attended tokens. That makes it vulnerable to exactly the attention misrouting Sun et al. describe, which turns it into a natural competitor and a natural test case.
  • CORA and XOR QA establish cross-lingual dense retrieval for single-hop questions. CORA's mDPR is the obvious retriever space to project into.

Research Program

The pipeline has four components:

  1. Model. An open multilingual decoder in the 7–13B range.
  2. Retriever. A frozen multilingual dense retriever in the style of CORA's mDPR, indexing a corpus that mixes English and native-language documents.
  3. Projector. An MLP from a chosen middle layer's hidden state into the retriever's query space, following L-RAG.
  4. Answering. Retrieve the first hop from the question embedding, append that document, run one forward pass, read the middle-layer state, project it, and retrieve the second hop. Repeat for longer chains. Only the final answer is ever decoded.

We first train the projector on English multi-hop data alone (HotpotQA, 2WikiMultihopQA, MuSiQue), with supervision from gold bridge documents. This English-only training is the strict tool-free condition.

The analysis stage asks where language-invariance breaks:

  1. Layer sweep. For each language and each layer, we measure the bridge-document recall that the English-trained projector achieves. This yields a "retrieval-by-layer" curve per language. In English the curve should peak in L-RAG's processing stage. Our prediction is that languages with larger closed-book gaps in Sun et al. show a lower or shifted peak.
  2. Head attribution. We locate the heads whose attribution scores mark them as routers in the retrieval setting, using their attribution-patching procedure.
  3. Intervention. We apply MFT-style tuning to those heads only, using closed-book XHOPREASON data, and re-run latent retrieval with the projector frozen. If bridge recall rises specifically in previously failing languages, we have causal evidence that the same broken routers break both closed-book reasoning and latent retrieval. That would mean one small repair serves both.

Baselines

Do nothing (single-shot multilingual retrieval). The question is embedded once by the multilingual retriever and the top documents go to the model. This is the floor that every multi-hop method must beat. For the cross-lingual multi-document case, the only reference point in the file is XRAG's GPT-4o score: 62% in English against 85% for humans, with relative drops above 50% in Arabic for some models (Liu et al. (2025)).

Native-language IRCoT. IRCoT interleaves chain-of-thought steps with retrieval, using each generated sentence as the next query. It improved MuSiQue retrieval recall by 11–21 points over one-shot retrieval. We run it in the query language with the same multilingual retriever, model, document budget, and prompt-tuning budget. It is the strongest accuracy competitor because it gets several full decoding passes. To beat it, latent retrieval must match its bridge recall within a few points at a fraction of its latency. Exceeding it in low-resource languages, where its generated queries are most likely to drift, would be a clear win.

Translate-test IRCoT. The question is machine-translated to English, IRCoT runs over the English portion of the corpus, and the answer is translated back. It breaks our tool-free constraint, so it is a reference point rather than a peer. It is still the baseline a practitioner would reach for.

DRAGIN. DRAGIN reports 0.3931 F1 on 2WikiMultihopQA with LLaMA2-13B, against 0.3076 for FLARE. This is the strongest existing internal-state method. We run its released triggering and query-formulation rules unchanged on our model and corpus.

Strongest reference numbers. For the latent method itself, L-RAG reports 0.352 accuracy on MuSiQue. We found no reported result on Lacuna for any iterative or latent multi-hop retrieval method on XRAG or on any non-English multi-hop retrieval benchmark. Our English runs must therefore first reproduce roughly the L-RAG and DRAGIN numbers before any cross-lingual comparison means anything.

Ablations. Two ablations isolate the key ingredient:

  • Projector training data. We compare the English-only projector against one trained with a small amount of target-language bridge supervision. This measures how much the method truly depends on language-invariance.
  • Representation used. We replace the middle-layer state with the final-layer state, and separately with the multilingual retriever's own embedding of the concatenated question plus first-hop document. The second variant tests whether the LLM's internal state adds anything over simply re-encoding the text.

Why not just…?

Why not just translate everything to English and use English multi-hop retrieval? This is a real threat for high-resource languages like German and Spanish, and we expect translate-test to be competitive there. Two facts in the file count against it as a general answer:

  • In XOR-Full the best translate-and-search baseline reached only 18.7 F1 (XOR QA), and CORA beat translate-test by 5.4 F1 on unseen MKQA languages (CORA).
  • About 20% of questions needed non-English evidence, which an English-only pivot cannot reach.

XRAG's multilingual setting puts some bridge documents in the native language, so we report results separately for hops whose gold document is native versus English. This should show where translation helps and where it caps performance.

Why not just run IRCoT in the user's language with a multilingual retriever? This is the most plausible competitor on accuracy, and it may win in high-resource languages. Our case rests on two things. First, latency: L-RAG's English measurements suggest roughly a sevenfold saving. Second, XRAG's documented drift into English during generation, which should corrupt generated queries. We measure directly how often IRCoT's intermediate queries come out in the wrong language. We then check whether its bridge recall falls in exactly those cases. If IRCoT matches latent retrieval in every language, we will report the latency trade-off as the only advantage.

Why not just use one long-context prompt with many retrieved documents? MultiHop-RAG shows that upfront retrieval with reranking reaches only 0.7467 Hits@10. Even with gold evidence, GPT-4 reaches 0.89 accuracy and Llama2-70B only 0.32 (MultiHop-RAG). XRAG locates the main cross-lingual penalty in synthesis over mixed-language documents, which more distractors should worsen. We include a top-50 long-context run at matched token budget to measure this directly.

Why not just fine-tune the whole model on multilingual multi-hop data? Sun et al. report MMLU drops above 60 points under DPO or full fine-tuning, against under one point for head-only tuning. We therefore keep full fine-tuning only as a cautionary comparison and measure MMLU and Belebele after every intervention.

Proposed Experiments

English reproduction. On HotpotQA, 2WikiMultihopQA and MuSiQue, we check that our latent pipeline lands near L-RAG's 0.352 MuSiQue accuracy and DRAGIN's 0.3931 2Wiki F1. MuSiQue matters most here because its construction resists single-paragraph shortcuts: 32.0 F1 for a single-paragraph model, against 64.8 on HotpotQA (MuSiQue).

Main cross-lingual evaluation. We use XRAG's Arabic, Chinese, German and Spanish questions over its English and mixed-language documents. The metrics are:

  • bridge-document recall at 1 and 5;
  • final accuracy;
  • response-language correctness;
  • wall-clock latency per query.

Controlled parallel test. We machine-translate MuSiQue questions into the XHOPREASON languages while keeping an English corpus. Every language then shares identical gold chains, so differences in bridge recall reflect the query language alone. We flag the translated-question artifact as a known limitation. XOR-TyDi QA serves as a single-hop sanity check that the retriever space is sound; CORA's 43.5 macro F1 is the reference.

Decisive checks. Two results would settle the core claim.

  1. Transfer. The English-only projector should retain most of its English bridge recall in at least the high-resource languages. If it collapses everywhere, the invariance claim is false, and the target-language-supervised ablation shows how cheaply the gap can be closed.
  2. Causal repair. Head-only repair should raise bridge recall in the languages whose layer curves sagged, with the projector frozen and MMLU within a point. If recall does not move, misrouting is not what breaks latent retrieval, and the closed-book mechanism does not extend to retrieval.

Either outcome is informative. The second check, applied to DRAGIN's text queries before and after repair, also tells us whether attention-guided query formulation inherits the same fault.