The standard recipe for teaching a language model to reason in Swahili, Tamil or Japanese is to take English reasoning data, such as grade-school math problems with worked solutions, machine-translate it, and fine-tune on the translation. This raises accuracy on multilingual math benchmarks. But every translated solution also contains names, places, dates and quantities rendered by a single translation system. The model is then trained to reproduce those renderings token by token. We propose to test a specific worry. As translated reasoning data grows in volume but stays narrow in how entities are written, fine-tuning may raise reasoning accuracy while quietly damaging the model's ability to state facts correctly in the target language. We call this damage entity drift. Our specific claim is that the damage sits in the final layers, where the model converts an internal, language-neutral concept into target-language words.
If this is right, reported gains in cross-lingual reasoning partly hide a rise in factual errors that nobody measures. The fix may also be cheap. Stop training the model to imitate translated entity strings (by removing those tokens from the training loss), and let its pretrained mapping from concepts to target-language words produce them instead. The note builds a controlled study of when entity drift appears as data volume and diversity vary. It localizes the drift inside the network and tests entity-span loss masking against the simpler fixes a reviewer would reach for first.
Core Claim
Three findings in the file motivate the hypothesis. Lost in Multilinguality uses the logit lens, which reads each layer's hidden state directly as a guess over the vocabulary. With it, the authors show that multilingual models usually retrieve the right fact in middle layers, in a roughly language-neutral "concept space," and then fail in the last two to four layers, where the concept is turned into target-language tokens. Bridging the Language Gap shows that non-English multi-hop factual failures come from attention heads routing the query to the wrong shared concept. It also shows that full fine-tuning and preference optimization used to repair such failures can drop MMLU by over 60 points. Dynamics of Instruction Fine-Tuning for Chinese LLMs finds that model-generated instruction data plateaus and declines earlier than human-curated data.
Taken together, these suggest a mechanism. Supervised fine-tuning (SFT) on machine-translated reasoning chains pushes gradient into exactly the fragile language-transition layers. It pushes them toward a narrow, sometimes wrong set of entity renderings, and this is a form of shortcut learning. The claim has three parts, and each is testable. First, entity-level factual accuracy in the target language falls as translated SFT volume grows, even while reasoning accuracy rises. Second, this falloff shrinks when the same data budget is spread over more languages. Third, the errors are mostly of the form "the correct entity appears in middle layers, the wrong entity appears at the output."
The predicted intervention follows from the mechanism. Entity-span masking keeps every translated reasoning step in the training loss but drops the loss on named-entity and date spans in the target-language solution. Reasoning is still learned by imitation. Entity rendering is left to the pretrained model.
Relation to Prior Work
The closest existing work is CCL-XCoT. It reduces multilingual hallucination with contrastive alignment on parallel text plus English-pivot chain-of-thought, and reports up to a 62% reduction in hallucination rates. Its authors state two limits: its effect on complex logical reasoning is limited, and it does not fix fine-grained errors in dates, numbers and names. It also treats its training data as fixed. Our proposal differs in three ways. It targets those entity-level errors directly. It keeps reasoning in the target language rather than pivoting through English. And it makes data volume and diversity the independent variable.
Beyond English-Centric Training shows that reinforcement learning (RL) with Group Relative Policy Optimization (GRPO) beats SFT on MGSM, and that RL models drift between languages mid-reasoning. It never measures factual fidelity. We borrow its training setup and add the missing measurement.
Three methods constrain or supply our design. Post-Training Language Models for Crosslingual Consistency introduces Direct Consistency Optimization (DCO), which aligns answer distributions across parallel prompts, but it is evaluated only on multiple-choice and short factual queries. Bridging the Language Gap introduces mechanistically-informed fine-tuning of about 40 attention heads. LinguaLIFT separates language alignment from English-only reasoning tuning. All three are candidate repairs, and we compare against them rather than build on them. We reuse the logit-lens readout of Lost in Multilinguality as our localization tool. We also reuse the per-layer weight-drift measurement of The Truth Stays in the Family, which measures how far each layer's weights move during fine-tuning.
Research Program
We fine-tune Qwen2.5-7B and Llama-3.1-8B, both used in the cited work, on machine-translated reasoning data from two sources. The first is translated GSM8K solutions, as in Beyond English-Centric Training. The second is multi-hop factual chain-of-thought built from a held-out training split of XHOPREASON's English queries, with the bridge entity written out explicitly in the solution. We then translate these into the eight XHOPREASON languages and a set of MMWP low-resource languages.
The experiment is a grid over two data axes. Volume takes four levels spaced by factors of four. Language diversity, at fixed total volume, spreads examples over one, four, or all target languages. Crossed with the data grid are four training treatments. Plain translated SFT is one. Entity-span masked SFT is another. The remaining two are simpler repairs: filtering examples whose entities fail a round-trip translation check, and SFT with the last four layers frozen.
Entity spans are found by aligning source and target named entities through the translation. Numerals are not masked, because intermediate values carry the reasoning. Instead, examples whose set of numbers changes under translation are dropped in all arms, so numeric noise is held constant.
Every checkpoint is scored on two kinds of measures. Reasoning is measured by final-answer accuracy on MGSM, MMWP and Cross-LogiQA. Factual fidelity is measured three ways. The first is target-language cloze accuracy on KLAR and BMLAMA, restricted to entities that never appear in the training data, so that we measure spillover and not memorization. The second is final-answer accuracy on XHOPREASON. The third is a new step-level metric: the fraction of XHOPREASON chains whose stated bridge entity matches the gold intermediate entity. On MultiHal's European-language subset, we also score open-ended answers with the benchmark's natural-language-inference entailment and HHEM-2.1 hallucination measures.
For localization, we apply the logit lens to every KLAR and XHOPREASON error. Each error is classified as either "right in the middle layers, wrong at the output" or "wrong throughout," and we track how the mix shifts with volume and treatment. Per-layer weight drift is reported alongside.
Baselines
The do-nothing reference is the untuned base model. Entity drift is defined relative to it: any checkpoint whose unseen-entity KLAR or BMLAMA accuracy falls below the base model's has lost facts.
The naive reference is plain translated SFT on all languages, equivalent to the Multi-SFT baseline in LinguaLIFT, which that paper reports at 39.4% low-resource accuracy on MMWP. It is the recipe the proposal claims is quietly harmful, so every other arm is compared against it at identical data, steps and learning-rate sweep.
The strongest reasoning competitor is GRPO trained on the same translated problems. Beyond English-Centric Training reports that its German-trained RL model averages 71.5% across ten MGSM languages and that RL beats SFT by 17.5 to 25.2 points. We run GRPO on Qwen2.5-7B with the same prompt set, and we report its compute separately so that accuracy per GPU-hour can be compared.
The strongest factual-consistency competitor is DCO applied after plain SFT, trained on parallel KLAR-style prompts. DCO is reported to raise BMLAMA non-English accuracy by up to 17.6% on Llama-3.1-8B. The strongest surgical competitor is mechanistically-informed fine-tuning of about 40 heads on XHOPREASON, reported at a 15% relative improvement with under one point of MMLU loss. On KLAR, the linear shortcut of Lost in Multilinguality reaches 76.08% on LLaMA2-7B; we reproduce it as an inference-time comparison for cloze recall only. For the step-level bridge-entity metric there is no reported result found on Lacuna, so all numbers there will be our own.
Round-trip filtering and final-layer freezing are the ablations that isolate the key ingredient, as explained in the next section. All arms receive the same tuning budget.
Entity-span masking beats the field if two conditions hold. At matched MGSM and MMWP accuracy, within one point of plain SFT, it must hold unseen-entity KLAR accuracy at or above the base model and beat plain SFT on bridge-entity fidelity. And it must do this without DCO's extra post-training stage or RL's sampling cost.
Why not just…?
Why not just filter out badly translated examples? This is the most serious threat, and we run it as a full arm. If entity drift is caused only by translation noise, filtering and masking will perform the same, and the right conclusion is "clean your data." Our mechanism predicts something stronger. Even correct translations push the final layers toward a narrow set of entity renderings, which harms entities never seen in training. The decisive comparison is on unseen-entity KLAR accuracy at high volume: masking should stay flat, while filtered SFT should still decline. If both stay flat, we report that filtering suffices.
Why not just freeze the last layers during fine-tuning? Freezing tests the localization claim directly. It is also cruder than masking, because the same layers must learn to write target-language reasoning. We expect freezing to reduce drift but also cost MGSM and MMWP accuracy. The mechanistic prediction is that it cuts "right in the middle, wrong at output" errors. If freezing matches masking on both reasoning and facts, it is the simpler method and we will recommend it.
Why not just use RL, which does not imitate translated text? Beyond English-Centric Training suggests RL may avoid this problem entirely, but nobody has measured its factual fidelity. RL's mid-chain language switching could lower entity drift, by relying on English pretraining, or raise it, by mixing entity forms across languages. We measure both. If GRPO matches masking on fidelity, masking's contribution becomes reaching similar fidelity at SFT cost, and we will state the compute ratio plainly.
Why not just reason in English and translate the answer? English-pivot reasoning, used in CCL-XCoT and XCOT, avoids target-language entities inside the chain. It still translates the final answer, however, and CCL-XCoT's authors note that entity-level errors remain. It also gives up target-language reasoning, which users of these languages may want. We include English-pivot chain-of-thought as an arm and compare final-answer entity accuracy.
Why not just repair consistency afterward with DCO? DCO is strong on the short multiple-choice queries it was built for. It has not been tested on open reasoning chains, and it adds a stage that depends on translations being reversible. Plain SFT followed by DCO is a direct comparison in our grid. If it fully restores fidelity, entity drift is still worth documenting, but prevention buys nothing over repair.
Decisive Checks
The first check is whether entity drift exists at all. Under plain SFT, unseen-entity KLAR and BMLAMA accuracy and bridge-entity fidelity should fall below the base model as volume grows, while MGSM and MMWP rise. If they do not fall, the core claim is false and the study ends as a negative result.
The second check is diversity. At fixed volume, spreading data over more languages should shrink the drift, consistent with the shortcut-learning account in Direction 3.
The third check is localization. The logit-lens error mix should shift toward "right in the middle, wrong at output" as drift grows, and weight drift should concentrate in the final layers. If errors are instead "wrong throughout," the damage is to routing, as Bridging the Language Gap would predict, and head-level repair becomes the better tool.
The fourth check is the filtering-versus-masking gap on unseen entities, which separates "noisy data" from "narrow imitation" as the cause. Together these four results tell practitioners whether translated reasoning data is trading facts for reasoning, where in the network the trade happens, and which of five available fixes addresses it at lowest cost.
References
- Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models
- Bridging the Language Gap: Uncovering and Aligning Shared Circuits for Multi-Hop Reasoning in Multilingual LLMs
- Dynamics of Instruction Fine-Tuning for Chinese Large Language Models
- CCL-XCoT: An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generation
- Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs
- Post-Training Language Models for Crosslingual Consistency
- LinguaLIFT: An Effective Two-stage Instruction Tuning Framework for Low-Resource Language Reasoning
- The Truth Stays in the Family: Enhancing Contextual Truthfulness via Inherited Heads in Model Lineages