Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization

Niyati Bafna, Kenton Murray, David Yarowsky

Abstract

While large language models exhibit certain cross-lingual generalization capabilities, they suffer from performance degradation (PD) on unseen closely-related languages (CRLs) and dialects relative to their high-resource language neighbour (HRLN). However, we currently lack a fundamental understanding of what kinds of linguistic distances contribute to PD, and to what extent. Furthermore, studies of cross-lingual generalization are confounded by unknown quantities of CRL language traces in the training data, and by the frequent lack of availability of evaluation data in lower-resource related languages and dialects. To address these issues, we model phonological, morphological, and lexical distance as Bayesian noise processes to synthesize artificial languages that are controllably distant from the HRLN. We analyse PD as a function of underlying noise parameters, offering insights on model robustness to isolated and composed linguistic phenomena, and the impact of task and HRL characteristics on PD. We calculate parameter posteriors on real CRL-HRLN pair data and show that they follow computed trends of artificial languages, demonstrating the viability of our noisers. Our framework offers a cheap solution for estimating task performance on an unseen CRL given HRLN performance using its posteriors, as well as for diagnosing observed PD on a CRL in terms of its linguistic distances from its HRLN, and opens doors to principled methods of mitigating performance degradation.

Introduction: The Challenge of Dialectal Variation

While Large Language Models (LLMs) like GPT-4 or BLOOM demonstrate impressive multilingual capabilities, they often struggle with "Closely-Related Languages" (CRLs) and dialects—such as Bhojpuri compared to Hindi, or Egyptian Arabic compared to Modern Standard Arabic. Investigating these failures is historically difficult due to data scarcity for low-resource dialects and the confounding factor of data contamination; it is often unclear if a model's success on a dialect stems from genuine generalization or accidental exposure during training.

Researchers from Johns Hopkins University have addressed this by decoupling these factors through the creation of "artificial dialects." By synthesizing languages with known, controlled linguistic distances from high-resource neighbors, they can "stress test" models and diagnose specific vulnerabilities.

Methodology: Controlled Linguistic Noising

The core of this framework involves parametrized probabilistic "noisers" ($\phi^n$) applied to a source high-resource language (HRLN). These noisers simulate four primary dimensions of language variation:

  • Phonological/Orthographic ($\phi^p$): Mimics sound changes and spelling shifts (e.g., $f \to v$) using character-to-IPA maps and context-aware rules.
  • Morphological ($\phi^m$): Simulates changes in word endings (suffixes) while keeping root stems intact.
  • Lexical - Function Words ($\phi^f$): Replaces high-frequency grammatical markers (e.g., "the", "and").
  • Lexical - Content Words ($\phi^c$): Replaces nouns and verbs with plausible non-words of similar length.
Table of noiser strategies and example transformations for phonological, morphological, and lexical variation.

The study measures Performance Degradation (PD), a normalized metric tracking the score drop relative to the "clean" HRLN and a random baseline. By calculating "posteriors"—estimating these noise parameters for real language pairs using bilingual lexicons—the authors can map real-world CRLs onto their artificial trends.

Experimental Setup

The study evaluated bloomz7b1 and mt0XXL across three task paradigms: X$\to$eng Translation (FloRes200), Commonsense Reasoning (XStoryCloze), and Natural Language Inference (XNLI). Source languages included Hindi, Indonesian, Arabic, German, French, and Spanish. Real-world validation was performed on CRLs such as Awadhi, Bhojpuri, Magahi, Maithili, Danish, Icelandic, Swedish, and various Arabic dialects.

Key Results and Insights

The researchers identified several critical trends in how LLMs handle linguistic variation:

  • Linear Degradation Trends: For translation tasks, PD follows a remarkably stable linear trend relative to noise parameters. This allows for the estimation of a model's performance on a completely unseen dialect simply by calculating its linguistic distance from a known neighbor.
  • The "Max" Effect: When combining different types of noise (e.g., lexical and morphological), the resulting performance drop is typically dominated by the most damaging single factor (usually lexical) rather than being additive.
  • Phonological Vulnerability: Small, consistent changes in spelling cause sharp performance drops, indicating that LLMs are brittle when the "visual" surface of the language shifts, even if the underlying logic remains intact.
  • Stem Robustness: Models are surprisingly adept at ignoring messy suffixes ($\phi^m$) as long as word stems remain recognizable, suggesting a heavy reliance on root-word semantics for Natural Language Understanding (NLU) tasks.
Performance degradation (PD) trends across tasks (Translation, NLI, StoryCloze) as a function of noise parameters, with real-world language pairs (CRLs) plotted for comparison.

Conclusion

This framework provides a principled method for diagnosing why an LLM fails on a specific dialect. By comparing real language performance to artificial trends, the authors even identified potential data contamination; if a real language (like Galician) performs significantly better than an artificial one with the same linguistic distance, it suggests the model likely encountered that language during training. Future work will look to incorporate syntactic variation and explore how few-shot prompting might mitigate these degradation curves.

Summary updated

Abstract

While large language models exhibit certain cross-lingual generalization capabilities, they suffer from performance degradation (PD) on unseen closely-related languages (CRLs) and dialects relative to their high-resource language neighbour (HRLN). However, we currently lack a fundamental understanding of what kinds of linguistic distances contribute to PD, and to what extent. Furthermore, studies of cross-lingual generalization are confounded by unknown quantities of CRL language traces in the training data, and by the frequent lack of availability of evaluation data in lower-resource related languages and dialects. To address these issues, we model phonological, morphological, and lexical distance as Bayesian noise processes to synthesize artificial languages that are controllably distant from the HRLN. We analyse PD as a function of underlying noise parameters, offering insights on model robustness to isolated and composed linguistic phenomena, and the impact of task and HRL characteristics on PD. We calculate parameter posteriors on real CRL-HRLN pair data and show that they follow computed trends of artificial languages, demonstrating the viability of our noisers. Our framework offers a cheap solution for estimating task performance on an unseen CRL given HRLN performance using its posteriors, as well as for diagnosing observed PD on a CRL in terms of its linguistic distances from its HRLN, and opens doors to principled methods of mitigating performance degradation.
BibTeX
@inproceedings{bafna2024evaluating,
  author = {Niyati Bafna and Kenton Murray and David Yarowsky},
  title = {{Evaluating Large Language Models along Dimensions of Language Variation: A Systematik Invesdigatiom uv Cross-lingual Generalization}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.1044},
}
Figure