Preventing Off-Target Translation in Multilingual Models

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2174 papers245 conceptsfrozen 2026-10-04

When you ask a Large Language Model (LLM) to translate a sentence from French to Spanish, you expect Spanish. But frequently, especially in "zero-shot" scenarios where the model hasn't been specifically trained on that pair, the model might respond in English, repeat the French source, or even provide a helpful meta-commentary in English about how it would translate the sentence if it could.

This phenomenon is known as off-target translation or language mismatch. It is a specific, structural form of hallucination where the model captures the semantic meaning of the task but fails to adhere to the linguistic constraints of the output.

The Tug-of-War Between Content and Constraint

To understand why off-target translation happens, think of the model’s decoding process as a tug-of-war. On one side is the content signal: the model’s understanding of the source text's meaning. On the other side is the language prior: the statistical likelihood of certain words following others based on the model's massive pre-training data.

In many-to-many translation systems, as noted by researchers at Huawei (HW-TSC), this problem scales with the number of languages. Because most models are trained heavily on English, the "English prior" is incredibly strong. When a model encounters a low-resource language or an unseen language pair, the language prior often overpowers the task instruction. As Marine Carpuat and Aquia Richburg observe, models frequently possess the underlying linguistic knowledge to translate but lack the "language grounding" to stay in the correct target lane.

Why Instruction Following Fails

Even when we give explicit instructions like "Translate the following to Swahili," models often fail. Haoran Xu and the team behind ALMA argue that this is a failure of cross-lingual instruction alignment. Base models might understand "Swahili" as a concept but haven't learned to strictly gate their output vocabulary based on that label.

This leads to several distinct failure modes:

  1. Source Language Persistence: The model gets "stuck" in the source language, essentially copying the input (Meta-XNLG).
  2. High-Resource Defaulting: The model translates the meaning correctly but defaults to English because it is the most "comfortable" path in its latent space (Alessandro Raganato).
  3. Meta-Talk Hallucination: Instead of translating, the model generates conversational filler or disclaimers, a problem Keqin Peng found is particularly common in non-English centric pairs.

Teaching Models to Reject the Wrong Language

If the problem is that the model is too "attracted" to the wrong language, one solution is to explicitly teach it what not to do. Changtong Zan and Liang Ding introduced a method called UNIONS (Unlikelihood Tuning on Negative Samples).

Instead of just training the model to predict the correct Spanish word, they also show the model "negative samples"—the same sentence in the wrong language—and use an unlikelihood loss to penalize those tokens. This effectively "pushes" the model away from the off-target language, forcing it to find the correct linguistic path. A similar approach, Language Aware Instruction Tuning, uses this unlikelihood strategy to reduce wrong-language outputs by over 50% (Changtong Zan).

Steering the Decoder

Another way to build intuition for this is through the lens of search. During decoding, the model explores a tree of possible word sequences. In a multilingual model, paths leading to English often have high transition probabilities simply because English is so prevalent in the training set.

To counter this, Stefan Lee and colleagues proposed Language-informed Beam Search (LiBS). This algorithm acts like a GPS for the decoder: it uses a separate, lightweight language identification model to check each branch of the search tree. If a path starts drifting into the wrong language, LiBS prunes it, ensuring the model stays anchored to the intended target. This demonstrates that off-target errors aren't just about "not knowing" the language; they are often a failure of the search process to maintain semantic meaning when a dominant language path is available (Language-Informed Beam Search Decoding).

Structural Hallucinations and Factuality

The consequences of language mismatch go beyond just getting the language wrong; they can actually distort the facts. Mingqi Gao and Xiaojun Wan found that structural disparities between languages (like how they handle verb tenses or plurality) can trigger "translation-induced hallucinations."

When a model is struggling to stay in the target language, it may incorrectly assign temporal or numerical attributes that weren't in the source text just to satisfy the grammatical requirements of the language it is trying to output. This suggests that linguistic consistency and factual accuracy are deeply intertwined in multilingual settings (Evaluating Factuality in Cross-lingual Summarization).

Summary of Solutions

The research direction has shifted from simply "adding more data" to more surgical interventions:

  • Better Signaling: Moving target language tags to the encoder to provide a stronger initial constraint (Liwei Wu).
  • Model Editing: Identifying and modifying specific "neurons" or attention heads responsible for language switching (Weichuan Wang).
  • Preference Optimization: Using cross-lingual alignment to ensure the model prefers the target language even when the content is complex (Xiaoye Qu).

By treating language adherence as a distinct constraint from semantic translation, researchers are making multilingual models not just more fluent, but more reliable.

Papers in this version

Showing 60 of 174 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 132 papers
  2. Version 1 · 2026-10-04 · 174 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_off_target_translation_6b0b4fd5,
  title = {off-target translation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/off-target-translation/rdr_6b0b4fd577194cdeab753bd2b91bce06},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_preventing_off_target_translation_in_mul_6b0b4fd5_v1,
  title = {Preventing Off-Target Translation in Multilingual Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/preventing-off-target-translation-in-multilingual-models/rdr_6b0b4fd577194cdeab753bd2b91bce06/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}