Abstract
We investigate a surprising limitation of LLMs: their inability to consistently generate text in a user’s desired language. We create the Language Confusion Benchmark (LCB) to evaluate such failures, covering 15 typologically diverse languages with existing and newly-created English and multilingual prompts. We evaluate a range of LLMs on monolingual and cross-lingual generation reflecting practical use cases, finding that Llama Instruct and Mistral models exhibit high degrees of language confusion and even the strongest models fail to consistently respond in the correct language. We observe that base and English-centric instruct models are more prone to language confusion, which is aggravated by complex prompts and high sampling temperatures. We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning. We release our language confusion benchmark, which serves as a first layer of efficient, scalable multilingual evaluation.
Introduction
As Large Language Models (LLMs) are deployed globally, their ability to communicate across diverse languages is paramount. However, a persistent failure mode exists: Language Confusion. This occurs when a model correctly identifies a user's intent but fails to adhere to the requested language, often defaulting to English. This research introduces the Language Confusion Benchmark (LCB) to systematically quantify and address this phenomenon.
Taxonomy of Language Confusion
Language confusion is not a monolithic error but manifests at three distinct levels of granularity:
- Full-response Confusion: The model generates the entire response in an incorrect language (typically English).
- Line-level Confusion: The model begins correctly but drifts into another language after one or more lines.
- Word-level Confusion: The model sporadically inserts foreign tokens into an otherwise correctly localized sentence.
This behavior increases tokenization costs for non-English scripts and degrades the user experience for non-English speakers.
The Language Confusion Benchmark (LCB)
To evaluate these failures, the researchers developed the LCB, covering 15 typologically diverse languages. The benchmark utilizes three primary metrics:
- Line-level Pass Rate (LPR): The percentage of responses where every line matches the target language.
- Word-level Pass Rate (WPR): The percentage of responses containing no "rogue" foreign words.
- Language Confusion Pass Rate (LCPR): The harmonic mean of LPR and WPR, serving as a unified reliability score.
Detection is performed via a pipeline using fastText for line-level identification and script-specific heuristics (e.g., Unicode range checks) for word-level errors.
Key Insights: Why Models Drift
The Instruction Tuning Trap
Counter-intuitively, instruction tuning can exacerbate language confusion. While base models often struggle with consistency, the instruction-tuned versions of Llama 2 and Llama 3 showed higher confusion rates than their base counterparts. This suggests that English-centric instruction tuning reinforces a bias toward English output, even when prompted in other languages.
Complexity and the "Confusion Point"
Model drift is more closely correlated with prompt complexity than prompt length. Multi-step reasoning tasks often trigger language switches. Analysis reveals that confusion typically occurs at "confusion points"—tokens where the probability distribution is relatively flat (high entropy). In these moments of uncertainty, the model is significantly more likely to sample an English token.
Results and Mitigation Strategies
Model Performance
Evaluation of state-of-the-art models revealed significant variance. Command R+ and GPT-4o demonstrated the highest robustness. Conversely, Llama 3 70B Instruct and Mistral models exhibited high confusion, failing to maintain language consistency in over 50% of certain monolingual tests.
Sampling and Few-Shot Prompting
Sampling parameters play a critical role. Lowering the Temperature (moving toward greedy search) significantly improves consistency. For instance, Command R's word-level pass rate improved from 86.5% at T=1.0 to 97.2% at T=0.0. Additionally, few-shot prompting (providing 1–5 examples) was found to nearly eliminate line-level confusion in base models by establishing a clear pattern of language adherence.
Limitations and Future Work
This study focuses on single-turn interactions and acknowledges that metrics may occasionally penalize natural code-switching (e.g., technical terms like "AI"). Future research will extend the LCB to multi-turn dialogues and refine detection for closely related Latin-script languages.
Summary updated