How do we know if a natural language processing model is performing well? We rely on automated metrics: BLEU or ROUGE for text generation, Exact Match or F1-score for question answering, Word Error Rate (WER) for speech recognition, and BERTScore for semantic similarity.
However, a fundamental flaw underlies almost all of these benchmarks: they assume the target language behaves like English. English is an isolating, analytic language with minimal morphology, rigid word order, and clear space-separated word boundaries. When these metrics are applied globally to languages with rich inflectional systems, flexible syntax, or complex compounding, their underlying mathematical and linguistic assumptions break down. As noted by Abhik Bhattacharjee et al., automated evaluation metrics frequently lack validity across diverse linguistic structures, particularly when applied outside of the high-resource languages they were originally optimized for.
The Morphological Penalty on Surface Matching
To understand why standard metrics fail, consider a simple task like extractive question answering. In English, if the gold-standard answer is "the cat" and a model outputs "cat", token-level metrics like F1-score penalize the model only slightly.
But in morphologically rich languages like Hebrew, prepositions, articles, and conjunctions are directly appended to nouns as prefixes or suffixes—a property often described as "sticky morphology." Amir Cohen et al. demonstrate that this creates a severe evaluation bias. Standard Exact Match and F1 metrics are too restrictive in these contexts; they fail to account for minor character-level variations that do not alter the underlying correctness of a retrieved answer, artificially depressing model scores. Similarly, in French reading comprehension, Martin d'Hoffschmidt et al. show that variations in average answer length and grammatical structure introduce a higher probability of boundary errors, making standardized metrics significantly harder to satisfy than in English.
English: to the house -> Three distinct tokens (High overlap potential)
Hebrew: לבית -> Single token with prefix (Zero overlap if character changes)
This surface-level brittleness is even more pronounced in Automatic Speech Recognition (ASR). Word Error Rate (WER) is the undisputed gold standard for English speech evaluation. However, in agglutinative languages like Malayalam or fusional languages like Arabic, a single word can contain multiple morphemes that would span an entire clause or sentence in English. If a model makes a single character error in a long agglutinated word, WER penalizes it as a completely incorrect word. To resolve this distortion, Thennal D. K et al. advocate for Character Error Rate (CER) as a more robust, universal metric that aligns far more closely with human judgment across diverse language families.
Preprocessing and Tokenization Volatility
One might expect modern, embedding-based metrics like BERTScore, MoverScore, or BaryScore to bypass these surface-level issues by operating in a continuous semantic space. Because they rely on contextualized representations rather than exact string matches, they should theoretically generalize across languages.
In practice, they remain deeply tethered to English-centric preprocessing pipelines. Jonas Belouadi et al. systematically investigated the reproducibility of these metrics and discovered that they exhibit severe volatility when applied to morphologically complex languages.
Because default tokenization and subword segmentation algorithms are optimized for English syntax, they fail to properly decompose complex inflections in other languages. As a result, an embedding-based metric's reliability varies wildly depending on subtle, undocumented preprocessing choices. The apparent "state-of-the-art" performance of these metrics often stems from hyperparameter tuning tailored to English, leaving their cross-lingual generalizability unverified.
Syntactic Flexibility and Language Density
Beyond word structure, evaluation metrics struggle to accommodate differences in syntax and word order. Traditional overlap metrics assume that word order is relatively fixed. In languages with highly flexible word order, such as German, a model can generate a perfectly natural and grammatically correct sentence that shares few n-gram overlaps with a single reference sentence. Aoife Cahill explored this phenomenon in German surface realization, finding that while automatic metrics can rank system outputs reasonably well, they fail to capture naturalness because they cannot accommodate German's grammatical flexibility.
A parallel issue arises when evaluating code-switching—the practice of alternating between two or more languages in a single conversation. Traditional code-switching metrics rely on raw token counts to measure the balance between languages. However, Jie Chi et al. point out that this approach fails to account for typological differences in language density. Because some languages naturally require more words than others to express the exact same concept, token-based metrics yield inconsistent measurements across diverse language pairs.
The Breakdown of Intrinsic Statistical Assumptions
When we move from translation and generation to evaluating intrinsic properties like linguistic complexity, fluency, or lexical richness, the English bias becomes an architectural constraint.
Many complexity metrics rely on features specific to English, such as syllable counts or dependency tree depth. As Jason Wei et al. note, these features do not translate effectively to morphologically rich or analytic languages, leaving a significant gap in cross-linguistic complexity measurement. Even fundamental statistical metrics like perplexity or bits-per-character are biased. Ryan Cotterell et al. demonstrate that bits-per-character is heavily skewed by the specific orthographic conventions of a language's writing system, making it an unreliable metric for comparing the inherent difficulty of language modeling across different languages.
Furthermore, referenceless fluency metrics like the Syntactic Log-Odds Ratio (SLOR)—which normalize log-probabilities from language models to evaluate text quality—have primarily been validated on English. As Katharina Kann et al. emphasize, it remains unproven whether these log-probability normalization techniques can successfully account for the complex grammatical structures and diverse word orders found in non-English linguistic systems.
Re-Engineering Metrics for Typological Diversity
To build evaluation frameworks that generalize globally, researchers must explicitly account for linguistic typology rather than treating non-English structures as "noise."
An early example of this adaptation is found in the work of Iryna Gurevych et al., who adapted Information Content metrics to German by introducing stemming to handle heavy inflection, significantly improving correlation with human judgments. More recently, researchers have proposed structural adaptations like the MaxMatch ($M^2$) algorithm for grammatical error correction (Daniel Dahlmeier et al.), which automatically aligns system outputs to human annotations by finding the optimal sequence of edits, though its generalizability to highly synthetic languages remains an active area of study.
Ultimately, the field is transitioning away from rigid, form-based string matching toward metrics that normalize for typological density, utilize character-level analysis, or incorporate language-specific morphological preprocessing. Without this shift, our evaluation frameworks will continue to suffer from an inherent cross-linguistic bias, misrepresenting model capabilities outside of English.
Key Foundational Work
Computing Semantic Relatedness in German with Revised Information Content Metrics (2005) — contributed 1 concept elements to this direction.
Syntactic Patterns in a Sample of Technical English (1969) — contributed 1 concept elements to this direction.