Small language models get better at answering questions that chain several facts together ("multi-hop" questions) when they are trained on large amounts of machine-generated question–answer data. That data has to be filtered first, because the generating models make mistakes. The standard filters keep an example only when two independent generators produce similar outputs. They measure similarity with surface overlap scores or sentence embeddings, and the cutoff for "similar enough" is set once, usually on English. We argue this cutoff is a hidden hyperparameter that encodes English's properties. In morphologically rich or low-resource languages, correct outputs share fewer surface tokens. In long reasoning chains, two correct solutions diverge more in wording. A fixed agreement threshold should therefore discard a disproportionate share of the deepest and most useful examples in exactly the languages where data is scarcest. We call this effect hop attrition.
The proposal has two steps. The first measures hop attrition directly, across languages and hop depths, with native-speaker audits of what the filters keep and throw away. The second replaces surface agreement with agreement on the reasoning chain itself. Two independent teachers must pass through the same sequence of intermediate "bridge" entities, linked to language-neutral identifiers, and each hop must be supported by a retrievable passage. Because this criterion ignores wording, it should keep deep, correct examples in any language while still rejecting hallucinated or shortcut-answerable ones. The test is whether students trained on chain-filtered data close more of the non-English multi-hop gap than students trained on data from the best-calibrated surface filter.
What Prior Work Establishes and Leaves Open
Three strands of prior work motivate the proposal.
Hyperparameters are not language-neutral. Configurations tuned on one language do not transfer to others:
- Improving Distributional Similarity with Lessons Learned from Word Embeddings showed that many gains credited to new architectures were really tuning effects.
- Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings found that peak accuracy required per-language tuning.
- Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages found that subword vocabulary sizes fail to transfer even within one language family.
These studies examine taggers, parsers, and translation systems. None asks whether the thresholds inside data filters carry the same bias, and none studies multi-step reasoning. That is the open question the file flags.
Agreement-based filtering works, but it has only been tested on shallow outputs. Several papers supply the filtering method we build on:
- Ensemble-Instruct keeps instructions on which heterogeneous open models agree under Rouge-L.
- Dual-Teacher Agreement for High-Precision Synthetic Data in Low-Resource MT combines surface, semantic, and fluency agreement between two translation teachers.
- Promptagator keeps a synthetic query only if a retriever can find its source document again (a "round-trip" check).
- TableLLM keeps a sample only if a text solution and a code solution agree.
Both the dual-teacher and Promptagator filters are documented to discard valid paraphrases and unusual forms. Multilingual Data Filtering using Synthetic Data from Large Language Models reports English-centric bias in LLM quality scorers. These are the known costs our hypothesis extends to reasoning depth.
Multilingual multi-hop reasoning is broken, and the closest existing fixes do not address data quality. - MuSiQue builds multi-hop questions by composing single-hop questions through bridge entities, and shows that models often answer without visiting intermediate hops. This supplies our construction recipe and our shortcut check. - Learning from Synthetic Data Improves Multi-hop Reasoning shows that rule-generated, verifiable-by-construction English data trains transferable multi-hop skill with outcome-only reinforcement learning. The method depends on fictional templated worlds and English only. - XCOT trains on machine-translated GSM8K without verifying the reasoning traces. - Bridging the Language Gap locates the non-English multi-hop failure in attention heads that misroute intermediate entities, and repairs them with targeted fine-tuning. It studies closed-book lookup and does not examine training data.
Our proposal differs from all of these because it works on the filter, and it targets natural, native-language, retrieval-grounded multi-hop data.
Core Claim
A fixed surface-agreement threshold produces hop attrition that grows with both linguistic distance from English and hop depth. Let $r_\ell(h)$ be the fraction of candidate examples at hop depth $h$ in language $\ell$ that survive a filter. We define
$$A_\ell(h) = \frac{r_\ell(h)}{r_\ell(2)},$$
the survival rate of $h$-hop examples relative to 2-hop examples. We predict three things:
- Under surface-agreement filters, $A_\ell(4)$ is markedly lower for languages such as Finnish, Swahili, and Telugu than for English.
- The gap in $A_\ell(4)$ is not explained by lower precision in the discarded examples, which native audits will check.
- A filter based on chain agreement keeps $A_\ell(h)$ roughly flat across languages at equal or better precision.
The prediction for surface filters follows from compounding. If each hop independently lowers the surface overlap of two correct solutions, overlap falls roughly geometrically with $h$. The per-hop penalty is larger when words carry more inflection and when the tokenizer splits them into more pieces, a mechanism documented in Targeted Multilingual Adaptation for Low-resource Language Families.
Research Program
Candidate pool. We build native-language multi-hop candidates in English, French, Chinese, Finnish, Swahili, and Telugu. These are all listed among the languages of MIRACL, whose native Wikipedia corpora we use for retrieval. Following the MuSiQue recipe, a generator writes single-hop questions from native passages. We compose them through shared entities into 2-, 3-, and 4-hop questions, so each candidate has a gold bridge-entity chain.
Teacher solutions. Two heterogeneous open teacher models each answer every candidate with interleaved retrieval in the style of IRCoT over the native corpus. Each teacher emits a reasoning trace, the passage it cites at each hop, and a final answer.
The chain-agreement filter. It has three parts:
- An entity linker maps each teacher's bridge entities to language-neutral knowledge-base identifiers. A candidate passes only if both teachers' identifier sequences match each other and the composition chain.
- Each cited passage must pass a per-hop round-trip retrieval check, adapting Promptagator from single queries to individual hops.
- A connectivity check, borrowed from MuSiQue's disconnected-reasoning idea, discards candidates that a single-paragraph model can already answer.
The filter compares structure, not wording. Its main hyperparameter, the entity-linking confidence, is far less tied to morphology than a surface-overlap cutoff.
Audit. Native speakers label a stratified sample of roughly 200 candidates per language and hop depth, both kept and discarded, for correctness of answer and chain. This lets us estimate each filter's precision and recall per cell instead of relying on downstream scores alone.
Students. Students are Qwen3 models up to 7B. Each is trained with a short supervised stage and then GRPO with outcome reward, following Learning from Synthetic Data Improves Multi-hop Reasoning, and uses iterative retrieval over the native corpora. GRPO (Group Relative Policy Optimization) is a reinforcement-learning method that scores each sampled answer against the others in its group. Each filter yields one training set, and every set is subsampled to the same number of examples and tokens per language.
Baselines
No filter. The do-nothing reference trains the student on the full unfiltered candidate pool, subsampled to the common budget. This tests whether outcome-reward reinforcement learning alone tolerates the noise. The dual-teacher MT study found that unfiltered synthetic data hurt: 41.8 chrF++ against 42.7 for parallel-only training, in Dual-Teacher Agreement.
Answer-only agreement. Teachers must agree only on the final answer. This is the ablation that isolates our key ingredient: if chain agreement adds nothing beyond answer agreement, the proposal fails.
Surface plus semantic agreement. This applies the composite filter of Dual-Teacher Agreement to the teachers' traces, run in two variants:
- an English-tuned threshold, as practitioners would deploy it;
- a per-language threshold chosen to match each language's overall retention rate to English, tuned with the same validation budget we give our own filter.
The calibrated variant is the strongest competitor, because it answers the obvious objection that hop attrition is merely a miscalibrated cutoff. On reasoning traces, the file contains no reported result for this filter; its only figure is the MT result above, 45.2 chrF++ after filtering.
LLM judge. An LLM scores each trace, after MDFS.
Translated rule-generated data. English PhantomWiki-style data is machine-translated into each language.
Translate-test pipeline. A translate-test system translates the question into English, runs English IRCoT, and translates the answer back. It needs no target-language training at all.
Head repair. For closed-book transfer, we compare against head-targeted fine-tuning from Bridging the Language Gap, which reports a 15% relative accuracy gain on XHOPREASON. The file gives no absolute value.
Criteria for beating them. All filters receive the same candidate pool, teacher outputs, training budget, and tuning budget. Our filter beats the field if it does both of the following:
- It improves 3- and 4-hop F1 on the non-English held-out sets over the calibrated surface filter, with confidence intervals across three seeds excluding zero.
- It holds English performance and audited precision within noise.
Why not just…?
Why not just tune the surface-agreement threshold per language? This is the most serious threat, and it is in the baseline set. Retention matching equalizes overall survival but cannot, by construction, equalize survival across hop depths within a language. The audit measures $A_\ell(h)$ under the calibrated filter directly. If the calibrated curve is as flat as ours at equal precision, the honest conclusion is that threshold calibration suffices, and we would report hop attrition as a calibration finding rather than claim a new filter.
Why not just translate verifiable rule-generated English data? Translated PhantomWiki-style data keeps answers verifiable. Learning from Synthetic Data Improves Multi-hop Reasoning reports 56% to 131% relative F1 gains in English from such data. However, its entities are fictional, so it never exercises retrieval over native corpora, where MIRACL shows large cross-language variation. The same baseline (BM25) scores 0.183 nDCG@10 on French and 0.551 on Finnish. It also cannot catch entity errors introduced in translation. The translated-data baseline settles this question empirically on native-corpus test sets.
Why not rely on outcome rewards, or answer agreement, to clean the data? Final-answer agreement accepts examples that both teachers answer through a shortcut. On English, MuSiQue shows how common such shortcuts are: single-paragraph models reach 64.8 F1 on HotpotQA but 32.0 on MuSiQue. We measure the shortcut rate of each training set with the single-paragraph probe, and the answer-only ablation shows whether shortcuts actually cost downstream multi-hop accuracy.
Why not translate the test question into English and reason there? XCOT reports that translate-then-reason prompting does not close the gap. An error in an entity at hop one also derails every later hop. Still, translate-test is cheap, so it remains a reference system. If it matches native students on the native-corpus test sets, the case for native-language training data weakens and we would say so.
Decisive Experiments
There is no existing open-domain, multilingual multi-hop benchmark in the file. For open-domain multilingual multi-hop QA over native corpora, Lacuna has no reported result. We therefore construct a held-out test set of about 500 questions per language, built MuSiQue-style from disjoint MIRACL passages and verified by native speakers. As external checks we also report English MuSiQue-Ans, where the strongest reported model reaches 47.3 F1 against 78.0 for humans, and closed-book transfer on XHOPREASON.
Three results decide the question:
- Existence of hop attrition. The audit must show that $A_\ell(4)$ under the surface filters falls with distance from English while precision among the discarded 4-hop examples stays high. Without this, there is no problem to solve.
- Equal precision, flatter retention. The chain filter must keep audited precision at the calibrated surface filter's level while flattening $A_\ell(h)$.
- Downstream gains where predicted. Students trained on chain-filtered data must gain most on 3- and 4-hop questions in Finnish, Swahili, and Telugu. A uniform gain across languages and depths would suggest the benefit comes from something other than reduced hop attrition.
A secondary analysis checks whether students trained on retained deep examples show less of the entity-misrouting seen by Bridging the Language Gap. This would connect the data-side and model-side explanations of the multilingual multi-hop gap.
References
- Improving Distributional Similarity with Lessons Learned from Word Embeddings
- Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings
- Optimal Word Segmentation for Neural Machine Translation into Dravidian Languages
- Ensemble-Instruct: Instruction Tuning Data Generation with a Heterogeneous Mixture of LMs
- Dual-Teacher Agreement for High-Precision Synthetic Data in Low-Resource MT
- Promptagator: Few-shot Dense Retrieval From 8 Examples
- TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios
- Multilingual Data Filtering using Synthetic Data from Large Language Models
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Learning from Synthetic Data Improves Multi-hop Reasoning
- XCOT: Cross-lingual Instruction Tuning for Cross-lingual Chain-of-Thought Reasoning
- Bridging the Language Gap: Uncovering and Aligning Shared Circuits for Multi-Hop Reasoning in Multilingual LLMs
- Targeted Multilingual Adaptation for Low-resource Language Families
- Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions