Imagine asking a massively multilingual large language model (LLM) a question in Swahili, only for it to reply with a factually flawless answer written entirely in English. Or imagine instructing a translation model to convert text from French to Spanish, but it simply outputs a modified version of the French input.
This phenomenon is known as off-target translation or off-target generation. It represents a fundamental breakdown not in a model's understanding of a prompt, but in its execution of the target language constraints. This issue is particularly acute in zero-shot scenarios—where a model translates between language pairs it never explicitly saw during training—and when dealing with low-resource languages.
To build reliable multilingual systems, researchers have had to look inside the black box of the Transformer architecture to understand why models lose their linguistic way, and how we can force them to stay on target.
Why Models Lose Their Linguistic Way
To understand why off-target translation happens, we must look at how multilingual models organize information internally.
Intuitively, a multilingual model performs a two-step dance: it maps an incoming language into a language-agnostic internal concept space, performs the necessary reasoning, and then maps that concept space back out to a specific target language. Wang et al. demonstrated this exact division of labor, showing that cross-lingual factual inconsistencies and language failures are frequently decoding errors occurring in the final layers of computation, rather than a lack of underlying knowledge.
This architecture creates two primary vulnerabilities that lead to off-target errors:
- The Gravitational Pull of High-Resource Languages: Because models are trained predominantly on high-resource languages like English, the pathways for these dominant languages are incredibly strong. Tang et al. discovered that LLMs possess "language-specific neurons" primarily located in the very first and very last layers, acting as routers. If the prompt fails to strongly activate the target language's specific neurons, the model defaults to its path of least resistance—the dominant training language (Lai et al.).
- Signal Decay and Token Entanglement: Standard architectures often signal the desired target language using a single token prefix (e.g.,
<2es>for Spanish). However, Liwei Wu observed that when target language signals are only provided at the beginning of a sequence, the decoder can lose track of this constraint as the generation grows longer. Compounding this, shared subword vocabularies can blur the boundaries between languages. As Liang Chen notes, when subword tokens are heavily shared across languages, the model lacks the discriminative signals needed to keep the output vocabulary isolated.
Architectural Isolation: Building Language Guardrails
One of the most effective ways to prevent off-target drift is to physically or logically separate the language pathways inside the model, preventing high-resource languages from overriding low-resource ones.
Instead of forcing a single shared parameter space to handle every language, researchers use modular architectures. Philip et al. demonstrated that replacing bilingual adapters with language-specific monolingual adapters significantly mitigates off-target errors in zero-shot translation. Taking this further, Pfeiffer and Nicosia introduced modular "bottleneck" modules that virtually eliminated source-language hallucinations, raising language consistency in zero-shot tasks from 7% to 99%.
Shared Semantic Space (Middle Layers)
│
├───► [English Adapter] ───► English Output
├───► [Spanish Adapter] ───► Spanish Output
└───► [Swahili Adapter] ───► Swahili Output (Protected from drift)
Other approaches focus on disentangling representations within the decoder itself. Zhi Qu, in a single-author paper, proposed a Central-Language-aware Layer (CLL) that explicitly separates language-specific information from universal semantics in the decoder, yielding massive improvements in zero-shot translation for low-resource languages. Similarly, Lei Li introduced a method to isolate unique language-specific subnetworks within a single model, reducing parameter interference and providing a stronger signal to distinguish between target language spaces.
Persistent Signaling and Vocabulary Splitting
If the root cause is a fading or ambiguous language signal, we can fix it by making the target language constraint louder and more persistent.
To prevent the target language token from being forgotten over long sequences, Jin and Xiong introduced Language Embedding Embodiment (LEE) and Language-Aware Multi-Head Attention (LAA). Instead of a single token at the start, these techniques inject richer, matrix-based language representations throughout every layer of the model, ensuring the target language anchor remains grounded during the entire generation process.
On the vocabulary side, Liang Chen tackled token entanglement directly with Language Aware Vocabulary Sharing (LAVS). By strategically splitting shared subword tokens into language-specific variants, the model is forced to learn highly discriminative boundaries between outputs, dropping off-target rates dramatically.
Inference-Time Steering
What if you cannot retrain or fine-tune the model? When working with frozen, off-the-shelf LLMs, researchers prevent off-target generation by intervening directly during the decoding process.
During standard beam search, an incorrect language path can occasionally accumulate high transition probabilities early on—especially if it drifts into a high-resource language like English. As the sequence grows, these off-target paths can outscore the correct target language path.
To counteract this, Stefan Lee developed Language-informed Beam Search (LiBS). This approach integrates a lightweight language identification model directly into the decoding loop.
Token Candidates ──► Language ID Classifier ──► Penalize Non-Target Tokens ──► Selection
As the model generates token candidates, the language classifier evaluates their linguistic identity in real time. If a path begins to drift into an incorrect language, LiBS applies a penalty to those tokens, effectively steering the beam search back toward the intended linguistic boundaries without requiring any structural modifications to the underlying model.