When Extra Context Makes Translators Pick the Common Meaning

Translation systems often get ambiguous words wrong in one predictable direction. They choose the meaning that was most common in their training data, such as "bank" as a financial institution, even when the sentence is about a river. Separately, language models are known to lose accuracy when irrelevant text is added to their input. This proposal asks whether these are two faces of one failure: does irrelevant context make a translator fall back on the most frequent meaning? If it does, we can fix part of the problem with a simple rule that needs no training. When adding context changes how a word is translated, accept the change if it moves toward a rarer meaning, and distrust it if it moves back toward the meaning the model would choose with no context at all.

The question matters because translation is increasingly done by large language models that receive extra text: retrieved passages, surrounding document sentences, and few-shot examples (sample translations placed in the prompt). Existing disambiguation fixes either retrain the model or depend on an external tagger that labels each word's meaning. Existing fixes for distraction were built for question answering and long-document retrieval, not for generating translations. Nobody has measured where noise sends ambiguous words. The working hypothesis is that it sends them toward the most frequent sense, and that this direction is itself a usable signal.

Core Claim

The claim has two parts. The first is empirical. Irrelevant context raises the rate of disambiguation errors, and it raises the share of those errors that land on the most frequent sense (MFS). Context whose vocabulary matches the dominant sense of the ambiguous word should do this more strongly than neutral filler. The second part is a method we call prior-directional arbitration. Frequency-bias errors run overwhelmingly in one direction. The DiBiMT benchmark found that nearly 90% of errors chose a more frequent sense than required, and that 93% of erring outputs gave the wrong translation higher probability (DiBiMT). Because of this asymmetry, a model can audit its own use of context. It does not need to know which context is relevant. It only needs to know whether the context pushed the translation toward or away from its own context-free default. Neither part requires a sense tagger, a sense inventory, or retraining.

Relation to Prior Work

Three papers motivate the hypothesis without testing it. Towards Effective Disambiguation for Machine Translation with Large Language Models prompts LLMs with example translations retrieved by a tagger to match the target word's sense. It finds that random examples stop helping, or start hurting, as their number grows from one to five. It never checks whether the extra errors are MFS reversions, and its method fails when retrieval finds no matching examples, which happened 7.5% of the time. Improving Word Sense Disambiguation in NMT with Salient Document Context prefixes each sentence with keywords extracted from its document. These short prefixes beat two-sentence full context on DOC-MUCOW. However, the method requires retraining, and adding prefixes at inference time to a model not trained for them degraded output. Exploiting WSD in LLMs for MT shows that listing every dictionary sense in the prompt does worse than giving the single predicted sense. It does not say whether the clutter caused MFS reversion or just confused the prompt format. Our first study answers exactly the question all three leave open.

Two lines of work supply the method's shape and also constrain it. Positional Contrastive Decoding rescues long-context answers by subtracting the output scores of a deliberately "near-sighted" copy of the model. It works because the correct token is usually still among the top few candidates. We borrow the idea of correcting decoding with a second forward pass of the same model. We differ in what the reference pass represents: the model's frequency prior, not a positional distortion. We also differ in when the correction applies: only when context moves the translation toward that prior. PCD is tied to rotary position embeddings and long-context retrieval, and was never tested on translation. Llama See, Llama Do locates a small set of attention heads that boost the scores of any token present in the prompt. It also finds that ablating those heads triggers backup circuits elsewhere in the network. That backup effect argues against a fix based purely on removing heads, and it supplies the mechanism we test: words tied to the dominant sense appearing in the context may be copied forward and strengthen that sense.

Two further papers bear on the claim. Adaptive Distraction shows that coherent distractors cut accuracy by about 45% on average and that telling the model to ignore irrelevant text does not prevent this. That is a negative result against instruction-based fixes. Contrastive Conditioning … Distilled Bias reports that distilling a large model into a smaller one amplifies MFS bias. This leads us to predict that distilled models will be the most prone to noise-induced reversion.

Research Program

Diagnostic benchmark. We build Noisy-DiBiMT from the 600 DiBiMT sentences across five language pairs. Each sentence is prepended with one of five context types, each about 1–3 sentences or k few-shot examples:

  • Neutral filler unrelated to the ambiguous word.
  • Dominant-aligned text, which uses vocabulary from the word's most frequent sense.
  • Rare-aligned text, which supports the required sense.
  • Mixed text, which is topically related to the sentence but also contains dominant-sense vocabulary.
  • Random parallel examples, following the setup of Iyer et al.

Context is generated by an LLM and then checked by annotators. This step ensures it does not alter the meaning of the target sentence. Dominant-aligned and mixed contexts are also checked so they do not imply the dominant sense for the target sentence itself.

The key measurement is the MFS shift. It is the change in the fraction of DiBiMT items translated with the most frequent sense, relative to the sentence-only condition. We report it separately for items whose gold answer is the MFS and for items whose gold answer is a rarer sense.

Separating two mechanisms. The comparison between neutral and dominant-aligned context separates two possible causes. In dilution, any extra text weakens the local cue and the prior wins. In entrainment, related words in the context actively push the dominant sense. On Llama-3-8B we additionally ablate the entrainment heads identified with the procedure of Llama See, Llama Do. We then check whether ablation removes the dominant-aligned excess but leaves the neutral-filler effect, as the entrainment account predicts.

Prior-directional arbitration. Decoding runs two passes: one with full context and one with the sentence alone. Where the two passes' top token choices differ (call them $a$ for the full-context choice and $b$ for the sentence-only choice), we consult a third, cheap reference. This reference is $\pi$, the model's probability for each candidate when it translates only the aligned source word, with no context. In the evaluation, the position of the ambiguous word is known. In deployment, an off-the-shelf word aligner finds it. The decision rule is

$$\tilde{L} = \begin{cases} L_{\text{full}} & \text{if } \pi(a) \le \pi(b) \\ (1-\lambda)L_{\text{full}} + \lambda L_{\text{sent}} & \text{otherwise,} \end{cases}$$

Here $L_{\text{full}}$ and $L_{\text{sent}}$ are the output scores (logits) of the two passes, and $\lambda \in [0,1]$ is tuned on held-out data. In words: context that moves the translation toward a less frequent reading is trusted, and context that moves it back toward the default is partly discarded.

The rule is applied only at positions where the two passes disagree, so the extra cost is small. The design has a known risk: it will harm items where the sentence alone wrongly picks a rare sense and the context correctly restores the MFS. We measure this cost directly on the MFS-gold items.

Baselines

Sentence-only translation. The naive reference point drops all context. If noise only ever hurts, this baseline is hard to beat on Noisy-DiBiMT. It should lose on DOC-MUCOW and on rare-aligned context, where context is genuinely needed. That is why both settings are included.

Full-context translation. The second naive point uses the full context unchanged. This is the setting whose MFS shift we measure.

Relevance filtering. This is the strongest training-free competitor. It adapts the passage scoring of Distraction-aware Answer Selection. That method weights retrieved documents by the probability that each document would generate the query, and it improved exact match by 64% on average in zero-shot question answering. We score each context chunk by the probability it assigns to the source sentence and drop low-scoring chunks. The threshold is tuned on the same held-out split and with the same number of forward passes as our method.

Contrastive decoding. The second strongest competitor amplifies the context effect: $(1+\beta)L_{\text{full}} - \beta L_{\text{sent}}$, in the form used by PCD, with $\beta$ tuned the same way. It tests whether the directional gate matters at all.

Confidence selection. This baseline picks whichever of the full-context and sentence-only outputs has higher probability.

Oracle-sense prompting. This is the strongest existing method with external help. It is sense-matched retrieval of examples via a tagger, following Iyer et al. The file reports that BLOOMZ 176B under this method beat DeepL by 7.3% on EN-ES and 12.2% on EN-ZH DiBiMT accuracy. It also reports DeepL's mean DiBiMT accuracy as 60.7% (DiBiMT). We cannot afford BLOOMZ 176B, so we rerun the method on our open models with the same example pool. No reported result was found on Lacuna for any disambiguation method under injected context noise, and none for keyword prefixing's absolute DOC-MUCOW F1.

Ablations. Two ablations isolate the key ingredient. The first applies the fallback to the sentence-only pass whenever the two passes disagree, regardless of the prior's direction. The second inverts the direction, trusting moves toward the prior.

What counts as success. Our method must beat relevance filtering and contrastive decoding on rare-sense accuracy in the mixed and dominant-aligned conditions. It must do so without falling below full context on DOC-MUCOW, and without losing more than one point of COMET22 on FLORES-200. Success also requires the undirected and inverted ablations to clearly underperform the directional rule.

Why not just…?

Why not just drop the context? On pure noise, dropping context will likely match us, and we expect that result and will report it. But the point of supplying context is the cases where it helps. Keyword-prefix gains concentrated on short sentences whose local cue is weak (Rippeth et al.). The rare-aligned condition and DOC-MUCOW measure what dropping context costs. If sentence-only translation ties our method there too, the method adds nothing and the contribution reduces to the diagnostic finding.

Why not just filter irrelevant chunks by relevance? This is a real threat. On neutral filler it may match us. It is also complementary: our gate could run after filtering. Its weak point is the mixed condition, where context is on-topic and therefore scores as relevant, yet carries dominant-sense vocabulary. Relevance scoring measures topic, not which sense the context pushes toward. If filtering equals arbitration on the mixed condition as well, we will conclude that the directional signal is unnecessary in practice and recommend filtering.

Why not just tell the model to ignore irrelevant text? Adaptive Distraction found such instructions ineffective against coherent distractors. The only instruction that did help in question answering offered an option to abstain, and a translator cannot abstain. We still include an instruction-prompt condition so the comparison is direct.

Why not just pick the more confident output, or plain contrastive decoding? DiBiMT's finding that 93% of erring outputs prefer the wrong translation by probability predicts that confidence selection will favor the MFS. That would make it worse than full context exactly where noise hurts. Plain contrastive decoding amplifies whatever context does. Under dominant-aligned context, our hypothesis predicts it will amplify the reversion. The inverted-direction ablation tests this logic directly.

Proposed Experiments

Models. We use Llama-3-8B (for continuity with the entrainment-head analysis), a larger open instruction-tuned LLM in the 65–70B range, BLOOM-7B, and a distilled translation student to test the distillation prediction.

Evaluation. The benchmarks are Noisy-DiBiMT across all five language pairs, DOC-MUCOW for the case where context is needed, HOLLY Korean–English as a second language family, and FLORES-200 with COMET22 and spBLEU to check that general translation quality is not harmed. We report DiBiMT accuracy, MFS-error share, MFS shift, accuracy on MFS-gold items, and extra decoding cost.

Decisive check 1: does the core claim hold? If dominant-aligned context does not raise the MFS shift above neutral filler, and neutral filler does not raise it above zero, the core claim fails. The method then has no basis, and we would report that noise causes unstructured errors rather than reversion to the frequent sense.

Decisive check 2: is the gain the directional gate? If our method's advantage over contrastive decoding and relevance filtering disappears under the undirected ablation, the gain comes from simply falling back to the sentence-only pass, not from the direction of the prior.

Decisive check 3: what does the gate cost? Accuracy on MFS-gold items under rare-aligned context measures the method's built-in downside. The method is worthwhile only if rare-sense gains clearly outweigh this loss in aggregate DiBiMT accuracy.