When a language model reads a retrieved passage that contradicts what it learned in training, it has to choose between the passage and its own memory. Recent multilingual benchmarks report that models usually side with the passage, but they score this choice by having the model pick from a short list of candidate answers. That list does more than record the decision. When the evidence is written in one language and the question is asked in another, the list also hands the model the target-language name of the entity it would otherwise have to produce itself. We propose to measure the same conflicts in three ways: the model picks from a list, the model writes the answer freely, and a linear probe reads the model's internal state. We then use these three readings to track what happens to conflict handling in non-English languages when the model is fine-tuned only on English task data.
The practical stakes are monitoring. Teams specialize multilingual models on English data constantly, and they check for damage with multiple-choice suites. If those suites cannot see the loss of cross-lingual conflict handling, a deployed retrieval system can quietly stop using non-English evidence. A useful test of the claim is whether the two scoring formats rank forgetting-mitigation methods differently. If they do, the multiple-choice numbers are not just noisy, they are misleading.
Core Claim
Our hypothesis has three parts. First, multiple-choice conflict scoring overstates how often the model follows the passage, and the overstatement is concentrated where the evidence crosses a script boundary, because the answer options provide the cross-lingual entity mapping. Second, fine-tuning on English erodes the model's ability to produce that mapping before it erodes its ability to recognize it. As a result, the persuasion rate measured by multiple choice stays roughly flat while the free-form rate falls. Third, the residual stream (the running internal vector each layer reads and writes) will show that the model still detects the conflict at the usual middle layers, and that what changes is the later layers that decide which answer to output.
Each part can fail independently. The third part is the mechanistic claim that would make the result more than a scoring complaint.
Positioning
The closest work is When Abundance Conceals Weakness: Knowledge Conflict in Multilingual Models. Its CLEAR framework measures how often models are stubborn (stick with memory) or persuaded (follow the passage) across 10 languages on ConflictQA-PopQA and ConflictQA-StrategyQA. It reports average persuasion of 81.0% and stubbornness of 13.4% on PopQA, and identifies a script barrier for cross-lingual evidence. We reuse its data and conditions. We differ from it in three ways: we rescore the same conflicts in free-form and probe formats, we add a fine-tuning stress test it does not run, and we look inside the model, which it does not.
Our suspicion that its high persuasion rate may partly reflect the format comes from the evaluation-bias literature. Can Multiple-choice Questions Really Be Useful in Detecting the Abilities of LLMs? finds Pearson correlations of only 0.33 to 0.70 between multiple-choice and long-form accuracy, and finds GPT-4 at 69% versus 58% on the two formats. It also documents a preference for earlier option positions. DOVE shows that surface changes such as replacing letter labels with numbers swing accuracy. All of this work concerns static English capability. None of it asks whether the format gap changes when a model is fine-tuned, which is the quantity a forgetting monitor depends on.
Our mechanistic readout borrows its method from two papers. Analysing the Residual Stream of Language Models Under Knowledge Conflicts shows that in Llama3-8B, linear probes detect conflict with about 90% accuracy at layer 14 and predict the context-versus-memory choice with AUROC above 0.85 by layer 17. That work is English-only and has no fine-tuning. Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions shows that the internal "knowledge" and "prediction" directions become nearly orthogonal in late layers under multiple choice. We test whether English fine-tuning widens a similar split specifically for non-English evidence.
The forgetting side is motivated by Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation. It shows that English-only tuning of mT5 makes the model forget how to generate in other languages, producing off-target or code-switched text. It reports that prompt tuning beat full fine-tuning by 7.3 SentencePiece-ROUGE points on Thai with mT5-XXL. That paper measures generation fluency in summarization. It does not measure conflict arbitration, and it does not compare scoring formats. Two papers motivate our expectation that arbitration is fragile under parameter updates: Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue and the head-level account of separate memory and context attention heads by Jin et al..
Research Program
Paired conflict instrument. From ConflictQA-PopQA we build items in which the same conflict is posed in three formats.
In the multiple-choice format, the CLEAR-style options include the memory answer and the passage answer. We permute option order and label style so that positional bias can be separated out.
In the free-form format, the model answers in the query language with no options. Because PopQA answers are entities, we score with multilingual alias lists for each entity rather than an LLM judge, which avoids the judge biases noted in the evaluation literature. Every free-form answer is assigned to one of five outcomes:
- matches the passage;
- matches memory;
- some other entity;
- refusal or hedge;
- answer in the wrong language, detected with language identification as in the cross-lingual generation paper.
Multiple choice cannot express the last three outcomes at all. Their combined rate is therefore a quantity the multiple-choice format cannot measure.
In the probe format, we train linear probes for conflict detection and for the context-versus-memory decision. Following the residual stream analysis, we train them on English activations and apply them unchanged to every language.
Every item is crossed with the CLEAR conditions: closed-book recall, evidence in the same language as the query, and evidence in a different language. Cross-script pairs are flagged.
Control set. ConflictQA-StrategyQA has yes/no answers, so multiple-choice and free-form scoring should nearly coincide there. If the format gap appears on StrategyQA as well, it is not about entity production, and our mechanism is wrong.
Fine-tuning stress test. We fine-tune Llama3-8B in two English-only regimes:
- a conflict-relevant regime, training for faithfulness to the passage on English NQ-SWAP and ConFiQA, which should raise English persuasion;
- a conflict-irrelevant regime, training on English MultiWOZ-style dialogue.
We take checkpoints along training. At each checkpoint we re-run the full instrument in all 10 languages, all three formats, and all evidence conditions. This gives a per-language, per-format forgetting curve, plus probe accuracy for each layer.
Baselines
The do-nothing reference is the untuned base model scored in all three formats. It sets the starting gap between formats, against which any change caused by fine-tuning is measured. Next to it sits plain full fine-tuning in each regime with no mitigation. This is the naive specialization we expect to reveal hidden forgetting.
The strongest existing preservation method for our setting is the recipe from Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation. It combines prompt tuning on a frozen model, mixing in 1% unlabeled target-language text, and factorized prompts that keep separate soft prompts for the task and for the language. It is the strongest competitor because it was designed for exactly the "English tuning destroys other languages" failure. Its reported gain is the 7.3 SentencePiece-ROUGE points on Thai with mT5-XXL. We know of no reported number for this recipe on any conflict benchmark.
Two further preservation methods are included because they attack forgetting differently:
- RECT from Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue, which keeps only the top 40% of weight changes by relative magnitude. We found no reported result on Lacuna for RECT on multilingual conflict data.
- Influence-selected rehearsal in the style of HESIT from Overcoming Catastrophic Forgetting by Exemplar Selection in Task-oriented Dialogue System, which reports 83.46% intent accuracy versus 78.22% for random rehearsal on a 37-domain English dialogue stream. We run it with a target-language validation set.
As a post-hoc repair we apply ContextFocus activation steering from ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models, using English-derived vectors. The paper reports needing as few as 1.5k examples. We found no reported numeric result on Lacuna for its faithfulness metrics that we could quote.
All methods share the same English training data and optimizer budget, with the same number of hyperparameter trials, tuned on English validation only. This keeps target-language test data unseen, as in the zero-shot setting.
The ablation that isolates our key ingredient is a multiple-choice variant in which the passage answer appears in the options only in the evidence language, not translated into the query language. If our mechanism is right, persuasion measured this way should drop toward the free-form rate on cross-script items. That would show the overstatement comes from the options supplying the translation, not from positional bias, which the permutation controls handle separately.
"Beating" multiple choice as a monitor means the following. The ranking of the four mitigation methods by forgetting should differ between multiple-choice and free-form scoring, with Kendall's tau clearly below 1 and confidence intervals from bootstrapping over items. The free-form ranking should also agree with the probe-based arbitration ranking.
Why not just…?
Why not just score everything free-form and drop multiple choice? For the final evaluation, that is roughly what we recommend if the hypothesis holds. The contribution is showing that this is necessary for this specific failure, because the existing multilingual conflict benchmark and most forgetting monitors use constrained formats. The StrategyQA control and the ranking-agreement test exist to establish where free-form scoring matters and where it does not. If multiple choice and free-form rank the methods identically, we will report that multiple choice is an adequate monitor here.
Why not just mix in 1% target-language data, as the cross-lingual generation paper does? This is a real threat. It may fully preserve conflict handling, in which case the forgetting we predict never appears. The experiment settles it directly, because this recipe is one of our baselines and is scored in all three formats. If it closes the free-form gap, the finding becomes that fluency-preserving rehearsal also preserves arbitration. That is useful, but it would demote the mitigation question. The scoring-format question still stands, since even then we can ask whether multiple choice would have detected the problem it fixed.
Why not just translate the query and the evidence into English at test time? GlobalWoZ reports up to a 60% failure rate when translated entity names are looked up against local knowledge, and below 10% joint goal accuracy for English-trained models on localized dialogue. Translation therefore reintroduces the entity-production problem in the translation step instead of removing it. We include translate-test as an extra row in the free-form evaluation so the cost is measured rather than assumed.
Why not just use the probe and skip behavioral scoring? Probes assume linearly readable knowledge, a limitation the knowledge-prediction paper acknowledges. Our hypothesis also predicts that detection survives while output fails, so a probe-only monitor would report that nothing is wrong. The probe is useful precisely as the third reading that tells us where in the model the failure sits.
Decisive Checks
The study is decided by four results.
First, on the base model, the gap between multiple-choice and free-form persuasion should be larger for cross-script evidence than for same-script evidence on PopQA, and near zero on StrategyQA.
Second, across fine-tuning checkpoints, the free-form persuasion rate for non-English evidence should fall while the multiple-choice rate falls much less. The wrong-language and other-entity outcomes should absorb most of the difference.
Third, probe detection accuracy around layer 14 should remain stable while the later arbitration probe degrades. Applying English steering vectors should recover English free-form behavior more than cross-script behavior.
Fourth, and most important for practice, the mitigation methods should be ranked differently under the two formats. Failure of the first check refutes the mechanism. Failure of the fourth while the others hold means multiple choice is biased but still usable for comparing methods, and we will say so.
References
- When Abundance Conceals Weakness: Knowledge Conflict in Multilingual Models
- Can Multiple-choice Questions Really Be Useful in Detecting the Abilities of LLMs?
- DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation
- Analysing the Residual Stream of Language Models Under Knowledge Conflicts
- Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions
- Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generation
- Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue
- Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models
- Overcoming Catastrophic Forgetting by Exemplar Selection in Task-oriented Dialogue System
- ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models
- GlobalWoZ: Globalizing MultiWoZ to Develop Multilingual Task-Oriented Dialogue Systems