Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluate

Ashim Gupta, Maitrey Mehta, Zhichao Xu, Vivek Srikumar

TL;DR

This paper introduces a simple yet highly effective framework called "Translate then Evaluate" to measure how consistently large language models (LLMs) respond to the same prompt across different languages. By evaluating 12 popular LLMs across 30 languages, the authors reveal severe cross-lingual consistency deficits in both factual information and empathy, particularly for low-resource language families and non-Latin scripts.

What Problem Does This Paper Address?

While state-of-the-art LLMs like GPT-4o and Llama-3 exhibit impressive capabilities in English, they are increasingly deployed globally in multilingual and multicultural contexts. A critical challenge is ensuring that these models behave predictably and equitably, regardless of the language a user speaks.

Standard multilingual evaluations rely heavily on expensive, human-annotated parallel datasets. Furthermore, evaluating open-ended generation tasks (such as long-form question answering or dialogue) is notoriously difficult because there is rarely a single "correct" ground-truth answer. As a result, models often provide rich, accurate responses in English but give truncated, superficial, or even flat-out incorrect answers when asked the exact same question in a lower-resource language. This disparity poses serious equity concerns and safety risks in high-stakes domains like healthcare and legal services.

Example of cross-lingual inconsistency in Llama-3 8B between English and Hindi responses

Methods

To bypass the need for expensive multilingual gold-standard data, the authors propose a Translate then Evaluate framework. The core idea is straightforward:

  1. Query the LLM with a prompt in a target language $\ell$ to get a response.
  2. Translate that response back into English using a high-fidelity translation system (specifically, the state-of-the-art NLLB 54B Mixture-of-Experts model).
  3. Use well-established, English-centric automated evaluation pipelines to compare the translated response against the model's original response to the same prompt in English.

The framework is instantiated across two distinct dimensions:

  • Information Consistency: Measures the factual overlap between responses using a modified version of FActScore. It breaks the text down into atomic claims and computes both precision (are the target language claims supported by the English response?) and recall (does the target language response capture the full information of the English response?).
  • Empathy Consistency: Uses the EPITOME framework to evaluate the presence of three therapeutic empathy mechanisms: Emotional Reactions (ER), Interpretations (IP), and Explorations (EX). The authors trained binary classifiers using ModernBERT-large to generate an empathy profile vector for both responses and checked for an exact match.

Datasets

The authors evaluated the framework using two primary prompt sets across 30 diverse languages spanning multiple scripts and language families:

  • Information Prompts: A manually curated dataset of 150 long-form, information-seeking prompts drawn from eight distinct categories of the MMLU benchmark (e.g., college chemistry, African history).
  • Empathy Prompts: 100 user utterances sampled from the test split of the Reddit subset of the EPITOME dataset, translated into the 30 target languages.

Key Insights and Ideas

  • The Script and Family Gap: LLMs are significantly more consistent when handling languages written in the Latin script compared to non-Latin scripts. The degradation is particularly severe for the Dravidian and Indo-Aryan language families.
  • Proprietary vs. Open-Weight Scaling: Stronger proprietary models (like the GPT-4o and Gemini 2.0 families) demonstrate much higher cross-lingual consistency than open-weight models. This suggests that cross-lingual alignment naturally improves as a byproduct of scaling overall model capability.
  • Consistency is Multi-Dimensional: A model that is highly consistent in conveying factual information may fail to maintain consistency in emotional tone or empathy, and vice versa. For example, the aya-exp-8b model showed poor information consistency but outperformed several proprietary models in empathy consistency.
  • Training Data Footprint: Models are vastly more consistent in languages that were explicitly highlighted in their official training documentation compared to out-of-distribution languages.

Key Results

The comprehensive evaluation yielded several striking quantitative findings (detailed in Table 4):

  • Top Performers: GPT-4o achieved the highest overall consistency, leading in both average information consistency (0.68) and empathy consistency (0.82).
  • Open-Weight Performance: Smaller open-weight models struggled significantly. For example, Llama-3.1-8B scored an average information consistency of just 0.32.
  • Dravidian Language Deficit: In the information consistency setting, aya-exp-8b dropped from a score of 0.59 on Latin-script languages down to a mere 0.06 for Dravidian languages.
  • Translation Reliability: To ensure the framework wasn't just measuring translation errors ("translationese"), a human study was conducted where native speakers rated translation fidelity. The system achieved a high average score of 4.26 out of 5 across ten tested languages, validating the framework's core assumption.
Averaged consistency scores across scripts, language families, and training data status

Limitations and Future Work

The primary limitation of this framework is its heavy reliance on an intermediate translation model. For the thousands of low-resource languages globally that lack high-fidelity translation tools, this framework cannot currently be applied. Additionally, while the authors successfully validated translation fidelity for factual information via human annotators, they could not do the same for the empathy dataset due to the scarcity of multilingual clinical experts.

Future work will focus on investigating the root causes of these low-resource language disparities and developing targeted alignment strategies to mitigate them.

Summary updated

Abstract

Large language models (LLMs) provide detailed and impressive responses to queries in English. However, are they really consistent at responding to the same query in other languages? The popular way of evaluating for multilingual performance of LLMs requires expensive-to-collect annotated datasets. Further, evaluating for tasks like open-ended generation, where multiple correct answers may exist, is nontrivial. Instead, we propose to evaluate the predictability of model response across different languages. In this work, we propose a framework to evaluate LLM's cross-lingual consistency based on a simple Translate then Evaluate strategy. We instantiate this evaluation framework along two dimensions of consistency: information and empathy. Our results reveal pronounced inconsistencies in popular LLM responses across thirty languages, with severe performance deficits in certain language families and scripts, underscoring critical weaknesses in their multilingual capabilities. These findings necessitate cross-lingual evaluations that are consistent along multiple dimensions. We invite practitioners to use our framework for future multilingual LLM benchmarking.
BibTeX
@misc{gupta2025found,
  author = {Ashim Gupta and Maitrey Mehta and Zhichao Xu and Vivek Srikumar},
  title = {{Found in Translation: Measuring Multilingual LLM Consistency as Simple as Translate then Evaluate}},
  year = {2025},
  eprint = {2505.21999},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.21999},
}
Figure