TL;DR
This paper introduces a simple yet highly effective framework called "Translate then Evaluate" to measure how consistently large language models (LLMs) respond to the same prompt across different languages. By evaluating 12 popular LLMs across 30 languages, the authors reveal severe cross-lingual consistency deficits in both factual information and empathy, particularly for low-resource language families and non-Latin scripts.
What Problem Does This Paper Address?
While state-of-the-art LLMs like GPT-4o and Llama-3 exhibit impressive capabilities in English, they are increasingly deployed globally in multilingual and multicultural contexts. A critical challenge is ensuring that these models behave predictably and equitably, regardless of the language a user speaks.
Standard multilingual evaluations rely heavily on expensive, human-annotated parallel datasets. Furthermore, evaluating open-ended generation tasks (such as long-form question answering or dialogue) is notoriously difficult because there is rarely a single "correct" ground-truth answer. As a result, models often provide rich, accurate responses in English but give truncated, superficial, or even flat-out incorrect answers when asked the exact same question in a lower-resource language. This disparity poses serious equity concerns and safety risks in high-stakes domains like healthcare and legal services.
Methods
To bypass the need for expensive multilingual gold-standard data, the authors propose a Translate then Evaluate framework. The core idea is straightforward:
- Query the LLM with a prompt in a target language $\ell$ to get a response.
- Translate that response back into English using a high-fidelity translation system (specifically, the state-of-the-art NLLB 54B Mixture-of-Experts model).
- Use well-established, English-centric automated evaluation pipelines to compare the translated response against the model's original response to the same prompt in English.
The framework is instantiated across two distinct dimensions:
- Information Consistency: Measures the factual overlap between responses using a modified version of FActScore. It breaks the text down into atomic claims and computes both precision (are the target language claims supported by the English response?) and recall (does the target language response capture the full information of the English response?).
- Empathy Consistency: Uses the EPITOME framework to evaluate the presence of three therapeutic empathy mechanisms: Emotional Reactions (ER), Interpretations (IP), and Explorations (EX). The authors trained binary classifiers using
ModernBERT-largeto generate an empathy profile vector for both responses and checked for an exact match.
Datasets
The authors evaluated the framework using two primary prompt sets across 30 diverse languages spanning multiple scripts and language families:
- Information Prompts: A manually curated dataset of 150 long-form, information-seeking prompts drawn from eight distinct categories of the MMLU benchmark (e.g., college chemistry, African history).
- Empathy Prompts: 100 user utterances sampled from the test split of the Reddit subset of the EPITOME dataset, translated into the 30 target languages.
Key Insights and Ideas
- The Script and Family Gap: LLMs are significantly more consistent when handling languages written in the Latin script compared to non-Latin scripts. The degradation is particularly severe for the Dravidian and Indo-Aryan language families.
- Proprietary vs. Open-Weight Scaling: Stronger proprietary models (like the GPT-4o and Gemini 2.0 families) demonstrate much higher cross-lingual consistency than open-weight models. This suggests that cross-lingual alignment naturally improves as a byproduct of scaling overall model capability.
- Consistency is Multi-Dimensional: A model that is highly consistent in conveying factual information may fail to maintain consistency in emotional tone or empathy, and vice versa. For example, the
aya-exp-8bmodel showed poor information consistency but outperformed several proprietary models in empathy consistency. - Training Data Footprint: Models are vastly more consistent in languages that were explicitly highlighted in their official training documentation compared to out-of-distribution languages.
Key Results
The comprehensive evaluation yielded several striking quantitative findings (detailed in Table 4):
- Top Performers: GPT-4o achieved the highest overall consistency, leading in both average information consistency (0.68) and empathy consistency (0.82).
- Open-Weight Performance: Smaller open-weight models struggled significantly. For example, Llama-3.1-8B scored an average information consistency of just 0.32.
- Dravidian Language Deficit: In the information consistency setting,
aya-exp-8bdropped from a score of 0.59 on Latin-script languages down to a mere 0.06 for Dravidian languages. - Translation Reliability: To ensure the framework wasn't just measuring translation errors ("translationese"), a human study was conducted where native speakers rated translation fidelity. The system achieved a high average score of 4.26 out of 5 across ten tested languages, validating the framework's core assumption.
Limitations and Future Work
The primary limitation of this framework is its heavy reliance on an intermediate translation model. For the thousands of low-resource languages globally that lack high-fidelity translation tools, this framework cannot currently be applied. Additionally, while the authors successfully validated translation fidelity for factual information via human annotators, they could not do the same for the empathy dataset due to the scarcity of multilingual clinical experts.
Future work will focus on investigating the root causes of these low-resource language disparities and developing targeted alignment strategies to mitigate them.
Summary updated