Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization

Niyati Bafna, Kenton Murray, David Yarowsky

Abstract

Large Language Models (LLMs) exhibit significant performance disparities across the world's languages. While high-resource languages (HRLs) are well-supported, closely related languages (CRLs) and dialects often suffer from substantial performance degradation. This paper presents a systematic framework for evaluating LLM robustness by modeling linguistic variation as a controllable Bayesian noise process. By perturbing HRLs along phonological, morphological, and lexical dimensions, we characterize the failure modes of state-of-the-art models and demonstrate that artificial noise trends can accurately predict model performance on real-world low-resource variants.

Introduction

Despite the impressive capabilities of models like GPT-4 and BLOOM, a "language gap" persists. Most training data is concentrated in a few dozen high-resource languages, leaving the majority of the world's 7,000+ languages underserved. Even when a model masters an HRL like Hindi, its performance frequently drops significantly on CRLs such as Maithili or Bhojpuri.

Quantifying this performance degradation (PD) is challenging due to data scarcity for dialects, potential training set contamination, and the lack of experimental control over the multiple linguistic shifts occurring simultaneously in real-world language pairs. We address these challenges by synthesizing "artificial dialects" where linguistic distance is a controllable variable, allowing for a granular diagnostic of model failures.

Methodology: Modeling Linguistic Variation

We treat linguistic variation as a Bayesian noise process applied to a source HRL. We define four specific "noisers" ($\phi^n$), each governed by a parameter ($\theta$) representing the probability of a linguistic change:

  1. Phonological/Orthographic ($\phi^p$): Simulates sound changes and spelling variations using character-to-IPA mappings and context-sensitive rules.
  2. Morphological ($\phi^m$): Corrupts word suffixes based on patterns prevalent in the target language family, simulating shifts in grammatical markers.
  3. Lexical - Function Words ($\phi^f$): Replaces high-frequency "glue" words (e.g., prepositions, articles).
  4. Lexical - Content Words ($\phi^c$): Replaces nouns and verbs with plausible-looking non-words of similar length to isolate vocabulary effects.
Table of noising strategies across phonological, morphological, and lexical dimensions with example inputs and model outputs.

By applying these noisers to HRLs, we generate a continuum of artificial variants. We validate this approach by estimating $\theta$ values for real-world language pairs (e.g., Spanish to Galician) using bilingual lexicons and comparing their PD to our artificial benchmarks.

Experimental Framework

We evaluated two prominent open-source multilingual models, bloomz7b1 and mt0XXL, across three diverse tasks:

  • Machine Translation (FloRes200): Translating the variant back into English.
  • XStoryCloze (XSC): Commonsense reasoning and story completion.
  • XNLI: Natural language inference (entailment vs. contradiction).

The framework was applied to seven source HRLs (Hindi, Indonesian, Arabic, German, French, Spanish, and English) and validated against dozens of real-world CRLs, including various Arabic dialects and Indo-Aryan languages.

Results and Discussion

Our systematic investigation yielded several key insights into the nature of cross-lingual generalization in LLMs:

Linear Degradation and Predictive Power

We observed a remarkably consistent linear relationship between the noise parameter $\theta$ and performance degradation across almost all tasks and languages. This linearity allows us to place real-world languages onto artificial noise curves with high accuracy. For instance, the PD of Maithili closely matches the predicted values based on its computed linguistic distance from Hindi.

Performance degradation (PD) as a function of noise parameters ($\theta$) across various tasks and languages, showing the linear trend and alignment with real-world closely related languages (CRLs).

Sensitivity to Linguistic Dimensions

Models proved highly vulnerable to phonological and orthographic shifts; even minor spelling variations can trigger sharp performance drops. Conversely, models showed surprising resilience to morphological noise. Even with 100% suffix corruption, models often successfully extracted core meanings from word stems. However, as lexical distance increases, vocabulary change becomes the dominant factor in model failure, following a "Max Principle" where the most damaging noise type dictates overall performance rather than an additive effect.

Task Robustness and Contamination

Machine Translation emerged as the most robust "stress test" for dialect understanding. Classification tasks like XNLI proved brittle, as noising specific label words (e.g., "Yes/No") can cause complete failure regardless of sentence comprehension. Furthermore, our framework serves as a diagnostic for data contamination; for example, Galician's unexpectedly high performance relative to Spanish suggested its presence in the model's training data.

Conclusion

This work provides a rigorous diagnostic tool for evaluating LLMs along dimensions of language variation. While models show some resilience to morphological shifts, they remain highly sensitive to phonological and lexical changes. Future research will expand this framework to include syntactic shifts and semantic "false friends," further closing the gap in our understanding of cross-lingual generalization.

Summary updated

Abstract

While large language models exhibit certain cross-lingual generalization capabilities, they suffer from performance degradation (PD) on unseen closely-related languages (CRLs) and dialects relative to their high-resource language neighbour (HRLN). However, we currently lack a fundamental understanding of what kinds of linguistic distances contribute to PD, and to what extent. Furthermore, studies of cross-lingual generalization are confounded by unknown quantities of CRL language traces in the training data, and by the frequent lack of availability of evaluation data in lower-resource related languages and dialects. To address these issues, we model phonological, morphological, and lexical distance as Bayesian noise processes to synthesize artificial languages that are controllably distant from the HRLN. We analyse PD as a function of underlying noise parameters, offering insights on model robustness to isolated and composed linguistic phenomena, and the impact of task and HRL characteristics on PD. We calculate parameter posteriors on real CRL-HRLN pair data and show that they follow computed trends of artificial languages, demonstrating the viability of our noisers. Our framework offers a cheap solution for estimating task performance on an unseen CRL given HRLN performance using its posteriors, as well as for diagnosing observed PD on a CRL in terms of its linguistic distances from its HRLN, and opens doors to principled methods of mitigating performance degradation.
BibTeX
@inproceedings{bafna2024evaluating,
  author = {Niyati Bafna and Kenton Murray and David Yarowsky},
  title = {{Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1044},
}
Figure