Abstract
Large Language Models (LLMs) exhibit significant performance disparities across the world's languages. While high-resource languages (HRLs) are well-supported, closely related languages (CRLs) and dialects often suffer from substantial performance degradation. This paper presents a systematic framework for evaluating LLM robustness by modeling linguistic variation as a controllable Bayesian noise process. By perturbing HRLs along phonological, morphological, and lexical dimensions, we characterize the failure modes of state-of-the-art models and demonstrate that artificial noise trends can accurately predict model performance on real-world low-resource variants.
Introduction
Despite the impressive capabilities of models like GPT-4 and BLOOM, a "language gap" persists. Most training data is concentrated in a few dozen high-resource languages, leaving the majority of the world's 7,000+ languages underserved. Even when a model masters an HRL like Hindi, its performance frequently drops significantly on CRLs such as Maithili or Bhojpuri.
Quantifying this performance degradation (PD) is challenging due to data scarcity for dialects, potential training set contamination, and the lack of experimental control over the multiple linguistic shifts occurring simultaneously in real-world language pairs. We address these challenges by synthesizing "artificial dialects" where linguistic distance is a controllable variable, allowing for a granular diagnostic of model failures.
Methodology: Modeling Linguistic Variation
We treat linguistic variation as a Bayesian noise process applied to a source HRL. We define four specific "noisers" ($\phi^n$), each governed by a parameter ($\theta$) representing the probability of a linguistic change:
- Phonological/Orthographic ($\phi^p$): Simulates sound changes and spelling variations using character-to-IPA mappings and context-sensitive rules.
- Morphological ($\phi^m$): Corrupts word suffixes based on patterns prevalent in the target language family, simulating shifts in grammatical markers.
- Lexical - Function Words ($\phi^f$): Replaces high-frequency "glue" words (e.g., prepositions, articles).
- Lexical - Content Words ($\phi^c$): Replaces nouns and verbs with plausible-looking non-words of similar length to isolate vocabulary effects.
By applying these noisers to HRLs, we generate a continuum of artificial variants. We validate this approach by estimating $\theta$ values for real-world language pairs (e.g., Spanish to Galician) using bilingual lexicons and comparing their PD to our artificial benchmarks.
Experimental Framework
We evaluated two prominent open-source multilingual models, bloomz7b1 and mt0XXL, across three diverse tasks:
- Machine Translation (FloRes200): Translating the variant back into English.
- XStoryCloze (XSC): Commonsense reasoning and story completion.
- XNLI: Natural language inference (entailment vs. contradiction).
The framework was applied to seven source HRLs (Hindi, Indonesian, Arabic, German, French, Spanish, and English) and validated against dozens of real-world CRLs, including various Arabic dialects and Indo-Aryan languages.
Results and Discussion
Our systematic investigation yielded several key insights into the nature of cross-lingual generalization in LLMs:
Linear Degradation and Predictive Power
We observed a remarkably consistent linear relationship between the noise parameter $\theta$ and performance degradation across almost all tasks and languages. This linearity allows us to place real-world languages onto artificial noise curves with high accuracy. For instance, the PD of Maithili closely matches the predicted values based on its computed linguistic distance from Hindi.
Sensitivity to Linguistic Dimensions
Models proved highly vulnerable to phonological and orthographic shifts; even minor spelling variations can trigger sharp performance drops. Conversely, models showed surprising resilience to morphological noise. Even with 100% suffix corruption, models often successfully extracted core meanings from word stems. However, as lexical distance increases, vocabulary change becomes the dominant factor in model failure, following a "Max Principle" where the most damaging noise type dictates overall performance rather than an additive effect.
Task Robustness and Contamination
Machine Translation emerged as the most robust "stress test" for dialect understanding. Classification tasks like XNLI proved brittle, as noising specific label words (e.g., "Yes/No") can cause complete failure regardless of sentence comprehension. Furthermore, our framework serves as a diagnostic for data contamination; for example, Galician's unexpectedly high performance relative to Spanish suggested its presence in the model's training data.
Conclusion
This work provides a rigorous diagnostic tool for evaluating LLMs along dimensions of language variation. While models show some resilience to morphological shifts, they remain highly sensitive to phonological and lexical changes. Future research will expand this framework to include syntactic shifts and semantic "false friends," further closing the gap in our understanding of cross-lingual generalization.
Summary updated