Understanding and Mitigating Language Confusion in LLMs

Kelly Marchisio, Wei-Yin Ko, Alexandre Bérard, Théo Dehaze, Sebastian Ruder

Abstract: We investigate a surprising limitation of LLMs: their inability to consistently generate text in a user's desired language. We create the Language Confusion Benchmark (LCB) to evaluate such failures, covering 15 typologically diverse languages with existing and newly-created English and multilingual prompts. We evaluate a range of LLMs on monolingual and cross-lingual generation reflecting practical use cases, finding that Llama Instruct and Mistral models exhibit high degrees of language confusion and even the strongest models fail to consistently respond in the correct language. We observe that base and English-centric instruct models are more prone to language confusion, which is aggravated by complex prompts and high sampling temperatures. We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning. We release our language confusion benchmark, which serves as a first layer of efficient, scalable multilingual evaluation at https://github.com/for-ai/language-confusion.

Introduction

As Large Language Models (LLMs) are deployed globally, their utility for non-English speakers is often hampered by a jarring user experience known as "language confusion." This occurs when a model understands a request but fails to provide the response in the desired language, often defaulting to English. While research has focused heavily on content-related errors like hallucinations, these "form" errors remain under-studied. For a model to be truly useful in a global context, it must respect the linguistic constraints of the user's environment.

Language confusion typically manifests at three distinct levels:

  1. Full-response: The entire output is generated in the wrong language.
  2. Line-level: The model drifts between languages across different sentences.
  3. Word-level: The model sporadically inserts English words into a non-English sentence.

The Language Confusion Benchmark (LCB)

To quantify these failures, we introduced the Language Confusion Benchmark (LCB), covering 15 typologically diverse languages. The benchmark draws from existing sources like Aya (human-generated), Dolly (machine-translated), and Okapi (synthetic), alongside newly commissioned native prompts in Japanese, Korean, Spanish, and French.

To test the limits of cross-lingual generation, we also developed "Complex Prompts"—long, multi-step English instructions that require the model to process information in one language and output it in another (e.g., "Summarize this English text in Turkish").

Metrics

We utilize off-the-shelf Language Identification (LID) tools, such as fastText, to measure performance via two primary metrics:

  • Line-level Pass Rate (LPR): The percentage of lines generated in the correct language.
  • Word-level Pass Rate (WPR): The percentage of individual words that match the target language.
  • LCPR: The harmonic mean of LPR and WPR, providing a single score for overall linguistic consistency.

Key Insights: The "English Gravity" and the Instruction Paradox

Our analysis revealed several surprising trends regarding why and when models fail.

The Instruction Tuning Paradox

Counter-intuitively, instruction-tuned models (such as Llama 3) often exhibit more language confusion than their base versions. This suggests that English-centric fine-tuning may inadvertently teach the model that "following instructions" is synonymous with "speaking English." While base models might struggle to follow the task, they often stay within the language of the prompt more consistently than their tuned counterparts.

Complexity vs. Length

Language confusion is not necessarily triggered by the length of a prompt, but rather by its complexity. Integrated instructions—where the task is buried in the middle of a text—are significantly harder for models to follow than instructions placed clearly at the beginning or end.

The "English Gravity" Phenomenon

By analyzing Shannon entropy and nucleus size during token generation, we found that confusion typically occurs at "confusion points" where the model's probability distribution for the next token is "flat" (high entropy). When no single token is a clear winner, the model's strong internal bias toward English—likely a result of English dominating the pre-training data—causes it to default to an English token, leading to language drift.

Experimental Results

Our evaluation across various model families showed a wide disparity in performance:

  • High Confusion: Llama 2, Llama 3, and Mistral models showed significant confusion, with LPRs as low as 46-48% in some monolingual settings.
  • High Stability: Models like Command R and GPT-4o were much more stable, maintaining LPRs above 98%.
  • Cross-lingual Challenges: Even the strongest models struggled with cross-lingual tasks. For example, while Command R+ achieved a 95.4% LPR on cross-lingual prompts, Llama 3 70B-I lagged behind at 30.3%.

Impact of Sampling

Inference-time settings play a major role in linguistic stability. Increasing the sampling temperature ($T$) significantly aggravates word-level confusion. Conversely, setting $T=0.0$ (greedy search) or employing beam search serves as an effective mitigation strategy.

Mitigation Strategies

Beyond inference-time adjustments, we identified several ways to reduce language confusion:

  • Few-Shot Prompting: For base models, providing 1 to 5 examples can dramatically improve performance, sometimes increasing accuracy from near 0% to over 90%.
  • Multilingual Fine-Tuning: Incorporating diverse linguistic data during Supervised Fine-Tuning (SFT) and preference tuning helps the model decouple the act of instruction-following from the English language.

Limitations and Future Work

While the LCB provides a critical first step, it currently focuses on single-turn interactions. Future work should explore multi-turn dialogues where language drift might accumulate over time. Additionally, current LID tools struggle to distinguish between closely related Latin-script languages (like Spanish and Portuguese) at the word level, and future benchmarks should aim to better handle natural code-switching used by multilingual speakers.

By identifying and quantifying language confusion, we hope to encourage the development of LLMs that are more reliable and respectful of the diverse linguistic needs of the global community.

Summary updated

Abstract

We investigate a surprising limitation of LLMs: their inability to consistently generate text in a user's desired language. We create the Language Confusion Benchmark (LCB) to evaluate such failures, covering 15 typologically diverse languages with existing and newly-created English and multilingual prompts. We evaluate a range of LLMs on monolingual and cross-lingual generation reflecting practical use cases, finding that Llama Instruct and Mistral models exhibit high degrees of language confusion and even the strongest models fail to consistently respond in the correct language. We observe that base and English-centric instruct models are more prone to language confusion, which is aggravated by complex prompts and high sampling temperatures. We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning. We release our language confusion benchmark, which serves as a first layer of efficient, scalable multilingual evaluation at https://github.com/for-ai/language-confusion.
BibTeX
@inproceedings{marchisio2024understanding,
  author = {Kelly Marchisio and Wei-Yin Ko and Alexandre Bérard and Théo Dehaze and Sebastian Ruder},
  title = {{Understanding and Mitigating Language Confusion in LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=Lgp11O38Sb},
}
Figure