When evaluating machine translation (MT) systems, the field has long relied on automated surface-level metrics like BLEU, ChrF, and TER. These metrics operate on a simple premise: the closer a machine-generated translation is to a human-authored reference text, the better its quality. In practice, this closeness is measured by counting overlapping sequences of words or characters (n-grams).
While this approach works reasonably well for analytic languages like English—where grammatical relationships are largely determined by word order and isolated helper words—it breaks down catastrophically when applied to morphologically rich languages (MRLs). For languages characterized by complex case systems, agglutination, polysynthesis, or extensive noun classes, standard evaluation metrics do not just fail to capture nuances; they systematically underestimate system performance and actively misguide architectural choices.
The Surface Matching Penalty
The fundamental flaw of exact-match metrics in morphologically rich contexts is their inability to recognize lexical and grammatical variations that humans consider valid or even identical in meaning.
Consider an agglutinative language where a single root word can take on dozens of different surface forms through the addition of prefixes and suffixes to denote tense, case, gender, or possession. If a machine translation model selects a grammatically correct or semantically appropriate variant of a word that differs even slightly from the specific form chosen by the human reference translator, an exact-match metric scores it as a complete failure.
As Nadir Durrani et al. observe in their work on Hindi-to-Urdu Machine Translation through Transliteration, these metrics penalize valid translations that use different word forms than the reference, heavily underestimating the system's actual utility. The authors of a study on Pre-Trained Multilingual Sequence-to-Sequence Models: A Hope for Low-Resource Language Translation? similarly demonstrate that BLEU heavily penalizes minor grammatical variations, masking the true capabilities of modern neural architectures.
This structural blind spot creates a severe misalignment with human judgment. In a report summarizing a Dagstuhl seminar on Statistical Techniques for Translating to Morphologically Rich Languages (Dagstuhl Seminar 14061), Alexander Fraser et al. emphasize that inadequate automatic metrics consistently underestimate translation accuracy for MRLs. This underestimation creates an artificial ceiling, making it difficult for researchers to fairly compare different models or track incremental progress.
The Downstream Consequences of Metric Bias
When engineering teams optimize machine translation models, they typically use automated metrics as a fitness function to select the best checkpoints or hyperparameters. When the fitness function is fundamentally biased against morphological complexity, it alters the nature of the generated language itself.
In Machine Translationese: Effects of Algorithmic Bias on Linguistic Complexity in Machine Translation, Eva Vanmassenhove et al. demonstrate that optimizing for standard benchmarks like BLEU or TER can lead to an artificial impoverishment of language, resulting in a simplified, homogenized output known as "machine translationese." Because complex morphological variations are risky and frequently penalized by exact-match metrics, models are implicitly incentivized to favor safer, simplified lexical choices. The authors note that the exact correlation between preserving linguistic diversity (such as morphological entropy) and maximizing traditional performance benchmarks remains a critical, unresolved tension in the field.
Furthermore, relying on flawed metrics can obscure critical errors that are immediately obvious to native speakers. Kshetrimayum Boynao Singh et al., in their evaluation of NITS-CNLP Low-Resource Neural Machine Translation Systems of English-Manipuri Language Pair (WMT 2023), point out that automated scores often fail to capture translation nuances and can mask structural errors. This sentiment is echoed by Yurii Paniv et al. in Setting up the Data Printer with Improved English to Ukrainian Machine Translation, confirming that relying solely on surface metrics fails to align with human evaluations of translation quality.
The High-Resource Generalization Gap
The inadequacy of standard metrics is compounded by a historical data imbalance: the tools used to evaluate the world's languages were designed and validated almost exclusively on a small handful of high-resource, Western languages.
As Sugyeong Eo et al. argue in QUAK: A Synthetic Quality Estimation Dataset for Korean-English Neural Machine Translation, the concentration of translation evaluation research on a few high-resource pairs prevents the development of evaluation tools that can generalize across varied linguistic structures. When these contemporary metrics are forced onto languages with entirely different typologies, their reliability vanishes:
- Polysynthetic and Agglutinative Languages: In Peru is Multilingual, Its Machine Translation Should Be Too?, Arturo Oncevay notes that word-level metrics fail to capture the semantic and structural nuances of highly agglutinative and polysynthetic indigenous languages like Aymara and Ashaninka. Similarly, Esther Ploeger et al., in Tokenization on Trial: The Case of Kalaallisut-Danish Legal Machine Translation, show that even when a subword tokenization strategy successfully preserves morphological boundaries in a polysynthetic language, this superior linguistic preservation does not translate to higher BLEU scores.
- Complex Grammatical Systems: Machel Reid, in AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languages, highlights how standard metrics fail to capture the intricacies of African languages that utilize complex noun class systems containing dozens of categories, making it impossible to automatically verify if a model is correctly applying grammatical prefixes.
- Dialects and Non-Standardized Languages: In Vuyko Mistral: Adapting LLMs for Low-Resource Dialectal Translation, Roman Kyslyi et al. demonstrate that traditional metrics inaccurately penalize valid dialectal variations, alienating the very communities the translation systems are meant to serve. Ona De Gibert et al. expand on this in the Findings of the AmericasNLP 2025 Shared Tasks on Machine Translation, Creation of Educational Material, and Translation Metrics for Indigenous Languages of the Americas, stating that surface metrics fail to capture semantic meaning and fluency for non-standardized Indigenous languages.
- Historical Languages: The problem extends to historical typologies as well. Eric R. Bennett et al., in Evaluating Evaluation Metrics for Ancient Chinese to English Machine Translation, show that metrics like BLEU and ChrF struggle with languages characterized by laconic phrasing, rich symbolism, and a lack of explicit grammatical markers.
Ultimately, as summarized in the Proceedings of the Eighth Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2025), the application of machine translation to low-resource and morphologically complex languages is severely impeded by this lack of dependable evaluation benchmarks.
Designing Robust Evaluation Frameworks
To break free from the limitations of exact n-gram matching, researchers are actively developing alternative evaluation paradigms that prioritize structural flexibility, semantic equivalence, and targeted linguistic probing.
Flexible Sequence Alignment
One approach is to replace rigid word-mapping with algorithms that accommodate reordering and morphological similarity. For instance, Samy Ouzerrout introduces UTER in UTER: Capturing the Human Touch in Evaluating Morphologically Rich and Low-Resource Languages. UTER incorporates a dedicated reordering algorithm and leverages the Sørensen-Dice similarity measure rather than strict identity matching. This allows the metric to tolerate natural variations in word forms and word order, aligning much more closely with human judgments of translation quality.
Targeted Morphosyntactic Probing
Instead of relying entirely on a single downstream translation score, researchers are also isolating specific grammatical capabilities using controlled perturbations. Vladislav Mikhailov et al. propose this paradigm via Morph Call in Morph Call: Probing Morphosyntactic Content of Multilingual Transformers. By evaluating how models handle guided sentence modifications across multiple languages, probing frameworks offer a controlled, nuanced assessment of how deep learning models represent and process specific grammatical features, moving beyond the limitations of standard text-generation metrics.
The Challenge for Modern Generative Models
The need for these robust frameworks has become even more urgent with the rise of Large Language Models (LLMs). As surveyed by Duygu Ataman et al. in Machine Translation in the Era of Large Language Models:A Survey of Historical and Emerging Problems, generative models produce highly fluent, varied, and complex outputs that traditional surface metrics are fundamentally unequipped to judge.
Whether evaluating an LLM attempting to automate grammar creation for endangered languages—as explored by Piyapath T. Spencer et al. in Can LLMs Help Create Grammar?: Automating Grammar Creation for Endangered Languages with In-Context Learning—or testing a metric's vulnerability to adversarial attacks, as investigated by Yichen Huang et al. in Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks, the conclusion remains the same: until our evaluation metrics can understand language beyond the surface level, our ability to develop translation systems for the full diversity of human speech will remain constrained.
Key Foundational Work
Statistical Techniques for Translating to Morphologically Rich Languages (Dagstuhl Seminar 14061) (2014) — contributed 3 concept elements to this direction.
Hindi-to-Urdu Machine Translation through Transliteration (2010) — contributed 1 concept elements to this direction.
Randomized Significance Tests in Machine Translation (2014) — contributed 1 concept elements to this direction.