Making Knowledge-Conflict Arbitration Independent of the Evidence's Script

When a language model is handed a document that contradicts what it memorized during training, it must decide whether to believe the document or its own memory. This is a knowledge conflict. The CLEAR study of such conflicts across ten languages (When Abundance Conceals Weakness) found something troubling. Whether a model is persuaded depends heavily on the language and writing system (script) of the evidence, and much less on what the evidence says. The same correct factual correction persuades the model when it arrives in Afrikaans but often fails when it arrives in Chinese, Japanese, or Bengali and the question is asked in a Latin-script language. The authors call this a "script barrier." Anyone building retrieval-augmented generation for users outside English will hit it. Retrieval-augmented generation (RAG) means a system that fetches documents and places them in the prompt. In a multilingual RAG system, the language a retrieved passage happens to be written in silently decides whether the passage is used.

This proposal has two stages. First, it finds where inside the model the script barrier arises. Second, it removes the barrier with a small intervention trained jointly across many language pairs. The intervention is taught to make a model treat foreign-script evidence the way it would treat the same evidence translated into the question's language. Translation is used only offline, as a teacher during training. At inference the model does one ordinary forward pass with no translation step. The goal is not to make the model more or less trusting of context. The goal is to make its trust independent of the evidence's script.

Core Claim

We claim that the script barrier is mostly a failure late in the forward pass, not a failure to notice the conflict. By "late" we mean the stage where the model copies the answer out of the context. The model can register that foreign-script evidence contradicts its memory. What it struggles to do is carry an entity name from a Bengali or Chinese passage into an answer written in English. Meanwhile the competing memory route produces the answer natively in the question's language. If this is right, three things follow.

First, a linear probe (a simple classifier trained on hidden states) should detect conflict about equally well for same-script and cross-script evidence at the middle layers. The probe's ability to predict whether the model will follow the context should diverge only in later layers.

Second, asking the model to answer in the evidence's language, rather than the question's, should largely restore persuasion.

Third, an intervention placed at the layers where extraction happens should close the gap. An identical intervention placed at the layers where conflict is detected should not.

A further claim concerns multilingual joint training. We claim that an intervention fitted on pooled conflict data from all language pairs will transfer to held-out languages and scripts. An intervention fitted on English-only data or on a single language pair will not. This mirrors the finding from cross-lingual understanding tasks that pooling languages beats pivoting through English.

Where this sits in prior work

The mechanistic picture comes from English-only studies, and these studies supply our measurement tools. Analysing the Residual Stream of Language Models Under Knowledge Conflicts showed the following for Llama3-8B. Conflict becomes linearly detectable around layer 14, with about 90% probe accuracy on NQ. The model's eventual choice between context and memory becomes predictable around layers 16–17, with AUROC above 0.85. Following context also produces a more concentrated activation pattern. Competition of Mechanisms and Cutting Off the Head Ends the Conflict showed that factual recall and context-following run on competing attention pathways, split between "memory heads" and "context heads," that meet in late layers. We reuse all three analyses, the probes, the concentration measures, and the head attribution, but run them on CLEAR's cross-lingual evidence condition. None of these studies examines non-English or cross-script inputs.

The motivating evidence comes from two papers. The first is CLEAR ( When Abundance Conceals Weakness: Knowledge Conflict in Multilingual Models), which documents the script barrier, the "query-memory congruence" effect, and the split between factual and reasoning conflicts. CLEAR does not look inside the model and does not try to fix anything. The second is Investigating Language Preference of Multilingual RAG Systems. It finds that generators ignore relevant non-Latin evidence even when there is no conflict. Its fix, DKM-RAG, is a translate-and-rewrite pipeline, and its authors themselves describe it as costly and dependent on translation quality. Our work differs from both. We study genuine contradictions rather than relevance, and we move the fix inside the model.

The intervention methods come from two steering papers. ContextFocus builds contrastive steering vectors, fixed directions added to hidden states, that push a model toward following context. It does so from as few as 1.5k examples, but it evaluates only in English. Improving Multilingual Language Models by Aligning Representations through Steering shows that a single middle-layer vector can pull non-English representations toward the model's English-aligned internal space. It reports a 26.7% improvement over supervised fine-tuning on MGSM and a 25.4% gain over NLLB-based translation. That work evaluates reasoning without any external evidence. We differ from both in the training target. Our target is the model's own activations when given translated evidence. It is neither "follow context more" (ContextFocus) nor "look more like English" (Mahmoud et al.).

Several papers constrain the design. Context versus Prior Knowledge in Language Models and When Not to Trust Language Models show that resistance to context grows with how familiar an entity is. Every comparison we make must therefore hold the entity fixed. The same question and the same evidence appear in different languages, so that entity popularity cannot masquerade as a language effect. The multilingual joint training literature supplies the hypothesis about pooling. It includes XGLUE (pooled fine-tuning beats English-pivot fine-tuning), Dhamecha et al. (related languages help low-resource relatives), and Alternating Language Modeling (code-switched training pulls embeddings together).

That literature also sharpens a puzzle, which CLEAR's findings appear to contradict. Joint training produces shared representations good enough for cross-lingual classification and parsing, yet arbitration between memory and context is not shared across scripts. Our diagnosis offers one resolution. Shared semantic space may exist at the middle layers where conflict is detected, while the copy-out route in late layers stays tied to the script.

Research Program

Stage one: diagnosis. We use CLEAR's ConflictQA-PopQA (898 entity questions) and ConflictQA-StrategyQA (1,000 reasoning questions), each in ten languages. For every item and every pair of question language and evidence language, we record hidden states from an open 8B model, starting with Llama3-8B so the English layer findings serve as a reference. From these we train three kinds of probes. Conflict probes and choice probes are trained per layer, following the residual-stream paper. Head-level attributions follow the memory-head and context-head method.

The key comparison is between same-script and cross-script evidence on identical items. For each layer we ask whether conflict detection survives the script change and whether choice prediction does. We also ask whether context heads still attend to the answer tokens in the evidence, and whether the late-layer boost to the answer token collapses. If the model attends to the right place but fails to raise the answer, extraction is failing.

Two behavioural tests support the probes. The first is the answer-language switch: the same prompt, but the model is told to answer in the evidence language. The second is a transliteration condition, in which evidence entity names are rewritten in the question's script and nothing else changes. This separates a barrier caused by the script from one caused by the language.

Stage two: a translation-distilled intervention. We fit a small low-rank adapter that edits the hidden state at a single chosen layer. The layer is picked from stage one, and a matched run places the adapter at the detection layer as a control. The notation is as follows. Let $h_\ell(q_A, e_B)$ be the last-token hidden state at layer $\ell$ for a question in language $A$ with evidence in language $B$. Let $T_A(e_B)$ be an offline machine translation of that evidence into $A$. The adapter $f_\theta$ is trained to minimize

$$ \mathcal{L}(\theta)=\sum_{(A,B)}\big\|h_\ell(q_A,e_B)+f_\theta(h_\ell(q_A,e_B))-h_\ell(q_A,T_A(e_B))\big\|^2+\lambda\sum_{A}\big\|f_\theta(h_\ell(q_A,e_A))\big\|^2 . $$

The first sum is taken over cross-script language pairs. It pulls the cross-script state toward the state the model would have had with translated evidence. The second sum is a penalty weighted by $\lambda$. It keeps the adapter nearly inactive when evidence is already in the question's language, so behaviour on same-language inputs is preserved.

Translation errors in the teacher are a real risk. Before training, we therefore filter to training pairs where the translated evidence still contains the counterfactual answer string. We report how often machine translation silently "corrects" a counterfactual entity back toward the familiar fact.

Stage three: joint versus pivot training of the intervention. We fit the adapter in four configurations. English-evidence-only pairs is the pivot condition. A single language pair, such as Chinese evidence with English questions, is the pairwise condition. The third configuration pools all pairs. The fourth pools all pairs except those involving one held-out language, and we rotate the held-out language across Bengali, Japanese, Swahili, and Icelandic. The held-out results test the joint-training hypothesis directly: pooling across scripts should yield an intervention that covers a script it never saw. Training data never overlaps test items. We build it from a split of the CLEAR items plus our own counterfactual edits of PopQA-style Wikidata facts, translated with NLLB.

Baselines

The naive reference point is the unmodified model on CLEAR's cross-lingual evidence condition. CLEAR reports an average Stubborn Rate of 13.4% and Persuasion Rate of 81.0% on ConflictQA-PopQA, and an average Stubborn Rate of 28.4% on ConflictQA-StrategyQA ( When Abundance Conceals Weakness: Knowledge Conflict in Multilingual Models). Stubborn Rate is how often the model keeps a correct belief despite contradicting evidence. Persuasion Rate is how often correct evidence fixes a wrong belief. We rerun this reference on our models rather than borrowing numbers, since CLEAR's models may differ from ours.

The strongest competitor is translate-test at inference. Every non-query-language evidence passage is translated into the question language with NLLB before the model sees it. This is the operation our adapter is trained to imitate, so it is effectively the teacher's upper bound. We run it with the same model and the same prompts. We add a second variant that translates everything into English, to test whether routing through English helps. Beating translate-test on accuracy is not required. The adapter counts as a success if it closes at least 80% of the gap between the naive model and translate-test on persuasion by cross-script evidence. It must do so in one forward pass, and it must not lose ground on items where translation mangles entity names. On low-resource evidence languages such as Swahili and Icelandic, we expect to beat translate-test outright, because translation quality is weakest there.

The strongest existing steering method is ContextFocus ( ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models), refitted on our pooled multilingual training pairs with its own recipe and the same example budget. The file reports only that it achieves "significantly higher" faithfulness than ContextDPO and COIECD on ConFiQA. No reported numeric result was found on Lacuna for ContextFocus on ConFiQA. ContextFocus is the natural rival because it also edits a hidden state with little data. It pushes uniformly toward context, however. We therefore expect it to raise persuasion but lower Stubborn Rate on items where the model's memory was right. Beating it means a comparable gain in cross-script persuasion without that drop.

We also include the English-alignment steering vector of Mahmoud et al. ( Improving Multilingual Language Models by Aligning Representations through Steering), applied at its recommended middle layer. It tests whether generic alignment toward English is enough. We further include Adaptive Contrastive Decoding ( Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts), which on noisy TriviaQA with Llama 2 7B reports 42.06 exact match against 24.75 for fixed-weight contrastive decoding. It represents the decoding-time family, at twice the inference cost.

All methods get the same training examples where they train, the same held-out test items, and a matched hyperparameter search of twenty configurations each on a shared validation split. The ablations isolate our key ingredients. We move the adapter to the detection layer. We replace the translation-matching target with a plain "follow context" target. We train the adapter on English-only pairs versus pooled pairs. We add a random-direction control of equal norm.

Why not just…?

Why not just translate the evidence at inference time? This is the real threat, and the proposal treats it as the teacher rather than dismissing it. DKM-RAG's authors report that translate-and-rewrite pipelines add heavy latency and inherit translation errors ( Investigating Language Preference of Multilingual RAG Systems). Mahmoud et al. found that internal steering beat NLLB translation by 25.4% on MGSM ( Improving Multilingual Language Models by Aligning Representations through Steering). Neither result is about knowledge conflict, though. Our experiments settle the question directly: translate-test is run as a baseline, and we separately report the rate at which translation damages counterfactual entities. If translate-test wins everywhere and rarely corrupts entities, the honest conclusion is that the contribution reduces to the diagnosis plus a cheaper single-pass approximation. We would report exactly that.

Why not just tell the model to use the evidence regardless of language? The TRUST-ALIGN authors ( Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse) found instructions of this kind brittle: models either ignore them or over-apply them. Resistance to context also grows with entity familiarity (Context versus Prior Knowledge in Language Models Context versus Prior Knowledge in Language Models). More fundamentally, if our diagnosis is correct, the barrier sits in the copy-out step. An instruction does not supply the missing ability to move a name across scripts. The prompted baseline tests this. The answer-language switch is itself a prompt-level fix, and if it alone removes the barrier, we will say that the simplest remedy is to answer in the evidence's language and then translate the short answer.

Why not just fine-tune on English conflicts and rely on zero-shot transfer? Pooled multilingual fine-tuning has consistently beaten English-pivot fine-tuning (XGLUE), and English-pivot reasoning leaves large gaps in Bengali and Thai until cross-lingual alignment is added (MAPO). Our English-only adapter configuration is exactly this alternative, so the comparison is inside the experiment. The held-out-script runs show whether pooling earns its cost.

Proposed Experiments and Decisive Checks

The main evaluation covers every pair of question language and evidence language in ConflictQA-PopQA and ConflictQA-StrategyQA. For each method we report three things. The first is Stubborn Rate and Persuasion Rate per pair. The second is a language-invariance gap: the absolute difference in persuasion between the same-script and cross-script versions of the same item. The third is the agreement rate between each method's answer and the answer the model gives with translated evidence. We also run CLEAR's multi-source task, where two contradicting passages arrive in different languages. There we check whether the adapter removes the bias toward high-resource evidence without collapsing the model's reasoning authority on StrategyQA. All numbers come with bootstrap confidence intervals over items.

For regression and transfer we use two further benchmarks. MKQA tests closed-book and open-book multilingual QA; it reports 46.0 macro F1 for translate-train XLM-R as a reference point for task difficulty. MzsRE tests cross-lingual knowledge editing. On MzsRE, ReMaKE-few-bi reports 69.26% reliability for English edits tested in all languages, with about 98% locality. We apply our adapter on top of ReMaKE's in-context edits, keeping its retriever and prompts fixed. Beating 69.26% reliability without dropping locality below ReMaKE's roughly 98% would show that the fix generalizes from benchmark conflicts to realistic cross-lingual updates.

Three results would decide the core claim. First, the diagnosis predicts that conflict-probe accuracy at the detection layer changes little between same-script and cross-script evidence, while choice-probe AUROC and context-head answer promotion drop sharply at later layers. If detection itself fails for cross-script evidence, the extraction hypothesis is wrong, and the intervention should move earlier. Second, the answer-language switch should recover most of the lost persuasion. If it does not, the barrier lies in comprehension rather than in copy-out. Third, the adapter placed at the extraction layer should outperform the identical adapter placed at the detection layer. The pooled adapter should also close the invariance gap on a held-out script such as Bengali, while the English-only adapter should not. If all three hold, we will have shown where the script barrier comes from and that a jointly trained, translation-free intervention removes it. If they fail, we will have shown that the barrier is not a late copy-out problem and pointed to where it actually is.

References

  1. When Abundance Conceals Weakness: Knowledge Conflict in Multilingual Models
  2. Analysing the Residual Stream of Language Models Under Knowledge Conflicts
  3. Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals
  4. Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Models
  5. When Abundance Conceals Weakness: Knowledge Conflict in Multilingual Models
  6. Investigating Language Preference of Multilingual RAG Systems
  7. ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models
  8. Improving Multilingual Language Models by Aligning Representations through Steering
  9. Context versus Prior Knowledge in Language Models
  10. When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories
  11. XGLUE: A New Benchmark Datasetfor Cross-lingual Pre-training, Understanding and Generation
  12. Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages
  13. Alternating Language Modeling for Cross-Lingual Pre-Training
  14. Improving Multilingual Language Models by Aligning Representations through Steering
  15. Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts
  16. Adaptive Contrastive Decoding in Retrieval-Augmented Generation for Handling Noisy Contexts
  17. Investigating Language Preference of Multilingual RAG Systems
  18. Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse
  19. Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse
  20. Context versus Prior Knowledge in Language Models
  21. MAPO: Advancing Multilingual Reasoning through Multilingual-Alignment-as-Preference Optimization
  22. MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering
  23. Retrieval-Augmented Multilingual Knowledge Editing