Many questions need two or more facts that sit in different documents, and in a multilingual world those documents are often in different languages. Take a Hindi question whose first fact is on English Wikipedia and whose second fact appears only in a Hindi or Japanese article. Today's systems answer such questions by retrieving one passage, reading it, and searching again. Each new search is driven by a vector that encodes both what the text is about and what language it is written in. We expect the second effect to pull later searches toward passages in the language of the current context. That drift would make the chain fail at exactly the hop that crosses a language boundary. This proposal builds a reader whose coarse decision ("which block of text is worth looking inside?") is trained to ignore language. Its fine decision ("which tokens inside that block carry the bridge entity?") is left free to use exact surface forms.
The mechanism adapts landmark attention. In landmark attention, a long context is cut into blocks. Each block gets one summary token, called a landmark, and the model attends to individual tokens only in proportion to how relevant their block's landmark is. We keep that two-level gate but train the landmark keys with a cross-lingual alignment objective borrowed from topic modeling. A block about the same subject then gets nearly the same landmark whether it is written in English, Hindi, or Japanese. The main claim is that alignment belongs at the coarse level and only there. Topic-level agreement across languages is achievable and useful for routing. Entity-level agreement is unreliable, and it is not needed once the coarse gate has chosen the right blocks.
Core Claim
Landmark Attention inserts a landmark token at block boundaries. Each token's attention weight is the product of a within-block score and the landmark's block-level score, so a block judged irrelevant is pruned whole. On LLaMA-7B this reaches 98% passkey accuracy at 32k tokens. Its landmarks are learned from a monolingual next-token objective and carry no pressure toward cross-lingual invariance. ProtoXTM shows the complementary lesson for the coarse level. Aligning languages through word-level dictionaries causes topic mismatch and degrades within-language interpretability. Contrasting document-level cluster prototypes instead improves cross-lingual topic coherence by up to 93.8%. ProtoXTM is a topic model, though; it never enters an attention layer or a reasoning chain.
We put these together. Write $q$ for the query state produced from a middle layer of the reader, $\ell_b$ for the landmark key of block $b$, and $k_i$ for the key of token $i$ inside block $b$. The token's attention weight is then
$$\alpha_i = \mathrm{softmax}_b\big(q \cdot \ell_b\big)\,\mathrm{softmax}_{i \in b}\big(q \cdot k_i\big).$$
During training, the landmark keys get an extra contrastive loss. Let $\ell_b$ and $\ell_{b'}$ be two blocks drawn from comparable articles in different languages, and let $\mathcal{N}$ be a set of other blocks used as negatives. The loss pulls these two keys toward a shared prototype $c$ of their topic cluster and pushes them away from the negatives:
$$\mathcal{L}_{\text{align}} = -\log \frac{\exp(\ell_b \cdot c / \tau) + \exp(\ell_{b'} \cdot c / \tau)}{\sum_{n \in \mathcal{N} \cup \{b, b'\}} \exp(\ell_n \cdot c / \tau)},$$
where $\tau$ is a temperature. The token keys $k_i$ get no alignment loss. The hypothesis is that this asymmetry, with an aligned coarse gate and an unaligned fine gate, closes the cross-language hop gap. We further expect it to do so without the topic-mismatch damage that ProtoXTM documents for lexical alignment, and without giving up the entity precision that multi-hop joins need.
Positioning Against Prior Work
Several papers motivate the gap. Investigating Information Inconsistency in Multilingual Open-Domain QA traced most cross-lingual answer inconsistency to the multilingual dense retriever picking unrelated passages, not to contradictory sources. Bridging the Language Gap used gradient-based circuit tracing to show that facts are stored in language-shared neurons, while specific attention heads misroute non-English multi-hop queries. It fixed this by fine-tuning only about 40 heads, which gave a 15% relative gain on XHOPREASON. Both papers point at routing, not knowledge, as the failure. Sun et al. study closed-book parametric reasoning only. We ask the analogous question for open-book reading over a multilingual evidence pool, and we intervene at the block-routing level.
Other papers supply parts of the method. L-RAG found that middle-layer hidden states are the best source of next-hop retrieval queries. It reports 0.352 accuracy on MuSiQue at roughly 2.8 seconds per query, against more than 20 seconds for iterative text-query methods. We reuse its idea of a middle-layer query, but aim it at aligned landmarks inside the reader rather than at an English Contriever index. Hierarchical Transformers for Multi-Document Summarization alternates attention within paragraphs and across paragraphs. It shows that this structure at 1,600 tokens beats flat models at 3,000 tokens (ROUGE-L 35.76 vs 34.90 for the previous best on WikiSum). That is evidence that the two-level inductive bias earns its place.
The closest systems all connect only two of the three directions. Hierarchical Lexical Graph has a topic tier for coarse filtering and an entity tier for joins, which is the same coarse/fine split we propose. It is an offline, monolingual English graph index built with costly LLM proposition extraction. HGRAG diffuses query relevance over an entity hypergraph whose hyperedges are passages, and it is likewise English-only. Cross-Lingual Multi-Hop Knowledge Editing is cross-lingual and multi-hop. It decomposes questions and retrieves single facts from a flat multilingual memory. It reports that weight-editing methods fall below 5% accuracy and that its retriever gains up to 30% over MeLLo and PokeMQA, but it notes large drops on non-Latin scripts. MIND routes across languages with polylingual topic clusters, but only for single-hop discrepancy detection.
None of these trains the reader's own block-level attention to be language-invariant while leaving token-level attention lexical. That combination is the contribution, and it is directly ablatable.
ProtoXTM also constrains the design. Its finding that entity- and word-level alignment produces mismatch is why we do not align token keys. We test that choice explicitly rather than assume it.
Research Program
The first deliverable is an evaluation set that actually separates hops by language. No existing benchmark in the file does this for open-book retrieval. MuSiQue is English. XOR-TyDi QA (XOR QA) is single-hop. CROLIN-MQUAKE and XHOPREASON are built around editing and closed-book reasoning. We will start from MuSiQue, because its single-hop composition and shortcut filtering make single-hop models lose about 30 F1 relative to HotpotQA. We will then relocate each hop's supporting paragraph into a target language.
Relocation will use the native-language Wikipedia paragraph when an interlanguage-linked article contains the fact, and machine translation otherwise. Every item is tagged with its provenance so translationese can be analyzed separately. Items are stratified along three axes: whether the hops cross a language boundary, whether the scripts are Latin or non-Latin, and whether the bridge entity's surface form is identical across the two languages. We will rerun MuSiQue's single-hop shortcut check on the relocated set to confirm that relocation has not reopened shortcuts. Distractors are drawn in all languages, so the query language does not give away where the answer lives.
The model is a multilingual 7B-scale decoder fine-tuned with landmark attention. Retrieved candidate passages are cached as key-value blocks, one passage per block, which avoids the fixed 50-token boundaries the landmark work uses. The landmark keys are trained with the prototype alignment loss. Aligned block pairs come from interlanguage-linked Wikipedia articles, treated as comparable documents in the polylingual-topic-model style. This avoids ProtoXTM's dependence on machine translation for positive sampling, which the file flags as fragile for low-resource languages. Hops happen inside a single forward pass. A middle-layer state, as in L-RAG, queries the landmarks, then the selected blocks are expanded to token level, and the bridge entity found there conditions the next landmark query. The candidate pool comes from one cheap multilingual first-stage retrieval with a generous budget.
Analysis centers on one diagnostic, which we call language-bias of block selection. It is the excess fraction of selected blocks that share the language of the current context, relative to the gold language distribution. We measure it per hop, separately for same-language and cross-language hops.
Baselines
The naive reference point is a single round of multilingual dense passage retrieval followed by the same reader with ordinary flat attention. This is the mDPR-plus-reader pipeline that Investigating Information Inconsistency diagnosed. It shows how much any multi-hop machinery buys at all.
The strongest existing method is IRCoT with a multilingual retriever. IRCoT alternates one-sentence chain-of-thought steps with retrieval, and it reports gains of 11–21 recall points and up to 15 QA F1 points on English multi-hop sets. It is the strongest competitor because it makes no topic or architectural assumptions and can, in principle, re-query in any language. We run two variants. One retrieves directly from the multilingual pool. The other is translate-test, which translates the question to English, runs IRCoT, and translates the answer back.
We also run L-RAG with its projection retargeted to a multilingual encoder. It is the most direct latent-retrieval competitor and the fairest latency comparison. We further run a CLEVER-CKE-style decompose-retrieve-verify pipeline over the same pool. That pipeline is the only cross-lingual multi-hop method in the file, and its reported advantage is up to 30% over MeLLo and PokeMQA on CROLIN-MQUAKE.
The file contains no reported result on Lacuna for any method on cross-language-hop MuSiQue, since the set does not yet exist. All of these systems will therefore be rerun by us. The external anchors are L-RAG's 0.352 MuSiQue accuracy and the 18.7 average F1 on XOR-Full. Every system shares the same backbone, the same first-stage candidate pool, the same retrieval and context budget, and an equal hyperparameter search budget of the same number of trials on the same development split. We report wall-clock latency on identical hardware.
Two ablations isolate the key ingredient. The first is the identical landmark reader trained without the alignment loss, which separates hierarchical gating from cross-lingual alignment. The second applies the alignment loss to token keys as well as landmarks, which tests ProtoXTM's mismatch claim inside our setting. A third variant uses the aligned landmark gate alone with no token-level expansion. It tests whether coarse topics can do joins, which HLG and HGRAG predict they cannot.
Beating the field means the following. On cross-language hops, the full model must beat both IRCoT variants on answer F1 and supporting-paragraph recall at no more than L-RAG-like latency. It must also beat the unaligned landmark ablation by more than the run-to-run spread on cross-language hops while staying within that spread on same-language hops.
Why not just…?
Why not just translate everything to English and run IRCoT? This is a real threat whenever the facts exist in English. XOR QA found that pivoting to English raised answer coverage by 22% on average, yet the full round-trip task still reached only 18.7 F1, and Investigating Information Inconsistency shows that locally anchored facts are often absent from English. Our stratification settles the question. We expect translate-test to stay competitive on items whose evidence has an English counterpart and to lose on native-only hops and non-Latin bridge entities, where translated names stop matching. If it wins across the board, the proposal's premise is wrong, and we will report that.
Why not just use a good multilingual dense retriever inside L-RAG or IRCoT? This is the most serious alternative, because nothing in the file directly measures language bias in multilingual retrievers during multi-hop chains. Our first decisive check is designed for exactly this. If an off-the-shelf multilingual retriever shows no cross-language hop-2 recall gap, the motivation disappears, and the work reduces to a latency study of landmark readers.
Why not run Hierarchical Lexical Graph's TopicGraphRAG with a multilingual encoder? It has the same coarse-then-fine shape. However, it depends on offline LLM proposition extraction, and its topic tier has no cross-lingual training pressure. We include it as an additional baseline on the smaller language pairs where its indexing cost is affordable.
Why not just feed everything into a flat long-context model? Bridging the Language Gap shows that flat attention heads misroute non-English multi-hop reasoning even when the facts are known, and the flat-versus-hierarchical comparison in Hierarchical Transformers favors structure over raw length. We still run the backbone with the whole candidate pool in its flat context as a control, at equal token budget.
Why not fine-tune the misrouting heads as Sun et al. did? That method targets closed-book recall, and it requires circuit discovery for each model. We test it as a complement: MFT-tuned heads combined with flat retrieval, compared against our reader. If MFT closes the open-book gap equally well, the simpler intervention wins.
Decisive Checks
Three results decide the project.
The first is the premise test. Before any training, does flat multilingual retrieval show lower hop-2 recall and higher language-bias when the gold hop-2 passage is in a different language from hop 1? We measure this on the relocated MuSiQue set, controlling for surface-form identity of the bridge entity. Without this gap there is nothing to fix.
The second is the mechanism test. At equal compute, the aligned-landmark reader should reduce language-bias of block selection and raise cross-language supporting recall relative to the unaligned landmark ablation. Same-language and English MuSiQue performance should hold. A further prediction is that aligning token keys too lowers accuracy on items with non-identical entity surface forms, which would reproduce ProtoXTM's mismatch inside attention.
The third is the transfer test. On CROLIN-MQUAKE we recast the multilingual edit memory as the block cache and ask whether the reader narrows the Latin versus non-Latin script gap that the CROLIN-MQUAKE authors leave open. On XOR-TyDi QA, as a single-hop sanity check, alignment must not hurt.
If the premise holds and the mechanism test passes, coarse language-invariant routing is a cheap fix that slots into existing landmark readers. If the mechanism test fails while the premise holds, the bias lives at the entity level, and that would redirect effort toward cross-lingual entity linking in the style of HIKE.
References
- Landmark Attention: Random-Access Infinite Context Length for Transformers
- ProtoXTM: Cross-Lingual Topic Modeling with Document-Level Prototype-based Contrastive Learning
- Investigating Information Inconsistency in Multilingual Open-Domain Question Answering
- Bridging the Language Gap: Uncovering and Aligning Shared Circuits for Multi-Hop Reasoning in Multilingual LLMs
- Optimizing Multi-Hop Document Retrieval Through Intermediate Representations
- Hierarchical Transformers for Multi-Document Summarization
- Hierarchical Lexical Graph for Enhanced Multi-Hop Retrieval
- Cross-Granularity Hypergraph Retrieval-Augmented Generation for Multi-hop Question Answering
- Cross-Lingual Multi-Hop Knowledge Editing
- Discrepancy Detection at the Data Level: Toward Consistent Multilingual Question Answering
- XOR QA: Cross-lingual Open-Retrieval Question Answering
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions