When evaluating a language model's performance across different languages, the most straightforward path is often machine translation: take an established English benchmark, translate it into the target language, and measure the model's accuracy. However, this process introduces a subtle but pervasive distribution shift known as translationese.
Translationese refers to text that, while grammatically correct, retains the structural, syntactic, or stylistic footprints of its source language. As the authors of The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation observe, translated benchmarks frequently suffer from target texts retaining unnatural artifacts from the original source. Even when the translation appears seamless, Duh et al. demonstrate in Is Machine Translation Ripe for Cross-Lingual Sentiment Classification? that the translation process introduces a systematic distributional bias that degrades downstream task performance.
This artifact distorts evaluation in two opposing ways: the Translation Penalty and the Circular Evaluation Trap.
The Translation Penalty
The translation penalty occurs when flaws in the translated benchmark text obscure a model's true reasoning capabilities. When a complex reasoning prompt or math word problem is translated, subtle shifts in phrasing can inadvertently alter its logical clarity.
In Mult-IT Multiple Choice Questions on Multiple Topics in Italian: A CALAMITA Challenge, the authors show that machine-translated evaluation datasets introduce unnatural grammatical structures and linguistic artifacts that interfere directly with the logic of multiple-choice questions. This creates a confounding variable: if a model fails a translated question, did it fail because it lacks reasoning skills, or because the prompt became linguistically incoherent?
As argued in Disentangling Text and Math in Word Problems: Evidence for the Bidimensional Structure of Large Language Models' Reasoning, translation artifacts confound our ability to isolate textual interpretation from core problem-solving. Li explicitly demonstrates in Marco-Bench-MIF: On Multilingual Instruction-Following Capability of Large Language that this penalty systematically underestimates large language models because the flaws in the translated instructions—rather than the model's reasoning—become the primary cause of failure.
The Circular Evaluation Trap
While the translation penalty underestimates a model's capabilities, an even more insidious effect can artificially overestimate them. Because native training data is scarce for many languages, developers frequently use machine-translated datasets to fine-tune models for multilingual instruction-following. When these models are evaluated on benchmarks that were also machine-translated from English, a dangerous loop occurs.
The authors of Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models? expose this circular evaluation bias. Because both the training data and the evaluation data share the exact same translationese artifacts, syntactic quirks, and English-centric knowledge patterns, the model appears highly competent. However, this competence vanishes when the model is tested on authentic, language-native text.
Furthermore, Etxaniz shows in Do Multilingual Language Models Think Better in English? that this translationese structure inherently favors models that translate inputs back into English internally, masking whether a model can actually reason natively within the target language's unique structure.
Erasing Culture and Nuance
Beyond syntax, translationese strips away the cultural and morphosyntactic context essential for true language understanding. Languages are not simple word-to-word mappings. As the authors of TUMLU: A Unified and Native Language Understanding Benchmark for Turkic Languages point out, translated benchmarks fail to capture the unique morphosyntactic structures and cultural nuances inherent to target languages.
This erasure extends to social dynamics; for example, the authors of The CRECIL Corpus: a New Dataset for Extraction of Relations between Characters in Chinese Multi-party Dialogues note that translated data fails to capture authentic social structures, such as specific familial honorifics and social hierarchies.
When evaluating abstract concepts like ethics, these subtle shifts can completely distort an evaluation. Farid et al. demonstrate in One Model, Many Morals: Uncovering Cross-Linguistic Misalignments in Computational Moral Reasoning that translation artifacts alter the cultural nuance or ethical framing of scenarios, making it difficult to accurately assess a model's true moral alignment.
The Shift to Native Benchmarks
To break free from the distortions of translationese, the field is moving toward natively authored benchmarks. Instead of translating English templates, researchers are sourcing data directly from local cultural contexts.
For example, the creators of ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic built their evaluation suite directly from local school exams rather than translating Western datasets. By eliminating translationese artifacts, these native benchmarks provide an uncorrupted view of how models handle authentic human language, ensuring that we measure true linguistic and logical comprehension rather than a model's ability to parse flawed translations.