Mismatch Responses in Multilingual Language Models

Multilingual language models often learn a task from English examples and then perform it, less well, in languages they were never trained on for that task. This is called zero-shot cross-lingual transfer. Why the drop happens is still unclear. The best current account comes from static comparisons: researchers feed a model the same sentence in two languages and measure how similar its internal states are, layer by layer. We propose a different probe, borrowed from auditory neuroscience's oddball paradigm. In that experiment a listener hears a repeated "standard" sound, then a rare "deviant," and the brain's response to the standard is subtracted from its response to the deviant. What survives the subtraction is a clean surprise signal, the mismatch negativity (MMN). The paradigm becomes powerful when two kinds of deviant are equally far from the standard physically but differ in category: one stays inside the same phonemic category, the other crosses a boundary. This contrast reveals which differences the brain treats as meaningful.

We apply the same logic in context. We show a model a stream of repeated sentences, then a deviant that changes only the language (same meaning) or only the meaning (same language), with the two deviant types matched on surface distance. Subtracting the model's states gives a layer-wise "mismatch response." It tells us where, and how quickly with repetition, the model stops treating a language switch as surprising. Our central bet is practical. This context-dependent signal should predict something that static similarity cannot: which in-context prompting format will transfer best for a given model and language, and whether a short "habituation" prefix of parallel sentences narrows the transfer gap.

Core Claim

The claim has two parts. First, a layer's matched-distance mismatch ratio is a label-free, context-dependent measure of how far the model treats language identity as a nuisance. The ratio compares the response to a language-only deviant with the response to a meaning-only deviant of equal surface distance, and we also track how fast that ratio falls as language switches are repeated. Second, because the measure depends on context, it can rank prompt formats and prefixes, which a single static similarity score per language pair cannot.

If the mismatch ratio merely reproduces what static similarity already predicts about transfer, the diagnostic half of the proposal fails. We say plainly below how we would detect that.

Grounding in Prior Work

The neuroscience side supplies the method. Processing of acoustic and phonological information of lexical tones in Mandarin Chinese revealed by mismatch negativity built an F0 continuum between Mandarin Tone 2 and Tone 4. It contrasted within-category and across-category deviants of identical acoustic step. Crossing a category shortened MMN latency from about 280 to about 250 ms, while step size affected only amplitude. The design separates "how different physically" from "how different categorically." Cortical Dynamics of Acoustic and Phonological Processing in Speech Perception used the same stimuli with fMRI. It found that across-category changes raise activity in left middle temporal gyrus and suppress activity in right primary auditory cortex. This is a layered picture in which higher regions encode category and dampen lower-level detail. We take from these papers the matched-distance contrast and the subtraction, and nothing about biology. The mapping from "acoustic step" to "language switch" is an analogy we test, not an assumption.

The language-model side motivates the proposal and also constrains it. Language-Specific Latent Process Hinders Cross-Lingual Performance is the closest existing work. Using CKA (centered kernel alignment, a representation-similarity score), cosine similarity, and logit lens on Gemma 2 and Qwen 2.5, it shows three things. Small models route non-English inputs through an English-like middle layer. Large models form language-specific paths. Cross-language layer similarity correlates above 0.90 with prediction consistency. Linear steering toward English helps small models but fails on large ones.

We differ from that paper in three ways. Our measure is conditioned on context, where theirs compares representations of isolated inputs. We control surface distance explicitly, which static similarity leaves confounded with conceptual distance. We track habituation over repetitions. Language-Specific Neurons Do Not Facilitate Cross-Lingual Transfer is a constraining negative result: editing or LoRA-tuning "language neurons" gives under one point on XNLI and XQuAD because of polysemanticity. We therefore measure whole-layer difference vectors rather than individual units. Cross-lingual QA: A Key to Unlocking In-context Cross-lingual Performance shows that keeping the passage in English while translating only the demonstration question-answer pairs beats both all-English and all-target prompts. That paper supplies the prompt-format comparison our measure should predict, and it offers no mechanistic account of why the format works.

Research Program

Stimuli. We build stimuli from parallel sentences, such as the 15-language XNLI premises from XNLI: Evaluating Cross-lingual Sentence Representations and BELEBELE passages. A block repeats a standard sentence $s$ several times, then presents one of two deviants:

  • a language deviant $d_{\text{lang}}$, the translation of $s$;
  • a meaning deviant $d_{\text{mean}}$, a different sentence in the same language as $s$.

The meaning deviant is chosen so that its distance from $s$ at the embedding layer matches that of the language deviant. This is the analogue of an equal acoustic step.

Controlling for position. A naive "deviant minus standard" confounds novelty with position. We therefore use an identity control. The response to sentence $x$ when it appears as a deviant is compared with the response to the same sentence $x$ when it is the repeated standard in another block, at the same position.

The mismatch ratio. At layer $l$, let $\mathbf{h}^{(l)}(x \mid \text{dev})$ and $\mathbf{h}^{(l)}(x \mid \text{std})$ be mean-pooled hidden states for $x$ in the two roles. The mismatch response is

$$\Delta^{(l)}(x) = \mathbf{h}^{(l)}(x \mid \text{dev}) - \mathbf{h}^{(l)}(x \mid \text{std}),$$

and the language-invariance ratio is

$$R^{(l)} = \frac{\mathbb{E}\,\lVert \Delta^{(l)}(d_{\text{lang}}) \rVert}{\mathbb{E}\,\lVert \Delta^{(l)}(d_{\text{mean}}) \rVert},$$

where expectations run over matched stimulus pairs. $R^{(l)}$ near zero means the layer barely registers a language switch relative to an equally large change of meaning. We compute a parallel version with next-token surprisal in place of hidden states. That gives a behavior-level prediction error.

Habituation. In a roving variant, the standards themselves alternate languages while the meaning stays fixed. We fit the decay of $\lVert\Delta^{(l)}\rVert$ over repetitions. The slope measures how quickly the model learns, in context, that language is irrelevant.

Hypotheses from prior findings. The English-pivot finding for small models predicts low middle-layer $R$ and fast habituation in Gemma 2 2B and small Qwen models. The silo finding predicts that large models (27B and up) keep a high middle-layer $R$ that habituates slowly. That would be a failure to suppress language identity across repetitions, which is the unexamined possibility raised in the research file.

Intervention. The mismatch measure also suggests a cheap test-time method: a habituation prefix of a few parallel sentence pairs placed before the task prompt. We test whether the prefix lowers $R$ and whether any drop in $R$ tracks the size of the accuracy gain.

Baselines

Naive reference points. These are plain zero-shot English-trained or English-prompted evaluation with no prefix and no intervention. For the diagnostic, the naive predictor of each language's transfer gap is the language's resource level together with the model's target-language perplexity on held-out text. Any structural measure must beat these, because data size alone may explain most of the variation.

Static representation similarity. This is the strongest diagnostic competitor: layer-wise CKA and cosine similarity between parallel sentences, as in Language-Specific Latent Process Hinders Cross-Lingual Performance, which reports a Pearson correlation above 0.90 with cross-language prediction consistency. We will compute it on exactly the same sentences, models and layers as our mismatch ratio, with the same layer-selection procedure chosen on held-out languages.

Beating it on its home ground is unlikely and is not our claim. The fair test is predicting which of three prompt formats wins per model-language pair, and the gain from the habituation prefix. Static similarity gives a single score per language pair, so for it we allow the best learned mapping from that score plus the format identity. We count as beating it a statistically reliable improvement in rank correlation, with partial correlation after controlling for CKA reported as the key number.

Strongest in-context transfer method. This is Cross-lingual QA prompting from Cross-lingual QA: A Key to Unlocking In-context Cross-lingual Performance. It reports 41.70 F1 for XGLM and 37.22 for BLOOM on XQuAD, against 35.19 and 34.56 for source-language prompting, with 7B-scale models. We rerun it on the same models, with the same number of demonstrations and token budget as our prefix condition. The habituation prefix counts as useful only if it adds F1 on top of Cross-lingual QA at matched prompt length.

Steering and neuron baselines. For the intervention question we include English activation steering with multipliers 2.0 to 3.0, as reported to help BELEBELE in small models. We also include the neuron-level results from Language-Specific Neurons Do Not Facilitate Cross-Lingual Transfer, where interventions rarely beat no intervention.

Ablations. Three ablations isolate the key ingredient:

  • dropping the matched-distance control, using an unmatched meaning deviant;
  • dropping the identity control, using naive deviant-minus-standard;
  • replacing the habituation prefix with an equal-length prefix of unrelated monolingual English sentences.

The third tests whether any prefix gain comes from the parallel, language-switching structure or simply from longer context.

Why not just…?

Why not just use CKA between parallel sentences, which already correlates above 0.90 with consistency? This is the real threat. It is why we do not rest the proposal on per-language prediction alone. Static similarity cannot change with the prompt, so it cannot on its own explain why Cross-lingual QA prompting beats full translation, or whether a prefix helps. Our measure can, and the prompt-format ranking experiment tests exactly this. If the partial correlation of $R$ with transfer after controlling for CKA is near zero and $R$ fails to rank formats better, then the oddball measure is a relabeling of CKA. We would report it as such.

Why not just translate the input into English at test time? Translate-test is strong. XNLI: Evaluating Cross-lingual Sentence Representations found it the best XNLI approach for most languages, with gaps of about 3 to 6 points. But it answers a different question: it bypasses the model's multilingual ability rather than explaining or improving it. Cross-lingual QA: A Key to Unlocking In-context Cross-lingual Performance also shows that fully translated prompts lose to selective translation on XQuAD. We report translate-test accuracy as an upper reference for the prefix intervention, not as a competitor to the diagnostic.

Why not just steer activations toward English? Steering helps small models and fails at 32B scale according to Language-Specific Latent Process Hinders Cross-Lingual Performance, and it risks pushing outputs toward English norms. Our habituation prefix changes only the input, and the mismatch measure lets us see whether it lowers language sensitivity in large models, where steering does not. If steering matches the prefix on small models and neither helps large ones, the prefix adds nothing and we will say so.

Proposed Experiments

Models. We use Gemma 2 at 2B, 9B and 27B and Qwen 2.5 across sizes up to 32B, following Language-Specific Latent Process Hinders Cross-Lingual Performance, plus XGLM and BLOOM at 7B for direct comparison with the XQuAD prompting results.

Mismatch curves. Stimuli come from XNLI's 15 languages and BELEBELE. We compute $R^{(l)}$ and habituation slopes for every model, layer and language, with at least several hundred matched pairs per language and bootstrap confidence intervals.

Downstream tasks. We evaluate on XNLI accuracy, BELEBELE accuracy, XQuAD F1 and MLQA F1. MLQA: Evaluating Cross-lingual Extractive Question Answering reports roughly 40% Exact Match degradation for XLM and F1 of 68.0 in Spanish, 54.8 in Arabic and 48.8 in Hindi, which gives a wide range of gaps to explain. Each task is run under source, target and Cross-lingual QA prompt formats, each with and without the habituation prefix.

First decisive check. Across held-out languages, does middle-layer $R$ predict the transfer gap with a reliable partial correlation after controlling for CKA and for resource level and perplexity? Does it predict, per model-language pair, which prompt format wins, at better than chance and better than CKA?

Second decisive check. Is the habituation prefix's F1 gain on XQuAD and MLQA, at matched length and on top of Cross-lingual QA prompting, correlated with the drop in $R$ that the prefix produces? Is the gain absent for the unrelated-English-prefix control? A gain without a matching drop in $R$ would mean the mechanism is not what we claim.

Third decisive check. Does the scale split in prior work reappear as slow habituation in large models? This would link the silo finding to a measurable failure to suppress language identity in context. The two ablations of the measure must weaken these effects. If removing the matched-distance or identity control leaves the results unchanged, the neuroscience-derived design was not doing any work.