The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?

Chen Shani, Yuval Reif, Nathan Roll, Dan Jurafsky, Ekaterina Shutova

Abstract

Multilingual language models (LMs) promise broader NLP access, yet current systems deliver uneven performance across the world's languages. This survey examines why these gaps persist and whether they reflect intrinsic linguistic difficulty or modeling artifacts. We organize the literature around two questions: do linguistic disparities arise from representation and allocation choices (e.g., tokenization, encoding, data exposure, parameter sharing) rather than inherent complexity; and which design choices mitigate inequities across typologically diverse languages. We review linguistic features, such as orthography, morphology, lexical diversity, syntax, information density, and typological distance, linking each to concrete modeling mechanisms. Gaps often shrink when segmentation, encoding, and data exposure are normalized, suggesting much apparent difficulty stems from current modeling choices. We synthesize these insights into design recommendations for tokenization, sampling, architectures, and evaluation to support more balanced multilingual LMs.

Introduction

Multilingual LMs like mBERT and XLM-R aim to provide universal NLP capabilities, yet they consistently perform better on high-resource, Latin-script languages than on others like Tamil, Arabic, or Turkish. This disparity raises a fundamental question: are some languages inherently harder for AI to model, or are we building models that are "English-centric" by design? If the difficulty is intrinsic, we may need new algorithms; if it stems from design choices like tokenization, we can fix the disparity through better engineering.

A Framework for Analyzing Multilingual Disparities

This study synthesizes research across six core linguistic properties: orthography, morphological complexity, lexical diversity, syntax, information density, and typological distance. By mapping these features to specific modeling artifacts, we can identify which design "levers" can be adjusted to improve fairness.

Table 1: Mapping linguistic factors to modeling artifacts and design levers.

Key Modeling Artifacts and Insights

The "Byte Premium"

Most models utilize UTF-8 encoding, where Latin characters typically require 1 byte, while scripts like Devanagari or Chinese can require 3 bytes. This "byte premium" means that within a fixed context window, the model effectively sees less information in non-Latin scripts, shrinking its functional memory and training signal.

Tokenization Fragmentation

Standard frequency-based tokenizers (like BPE) are often optimized for English-like structures. In morphologically rich languages, these tokenizers "shred" complex words into semantically meaningless fragments. Research indicates that when morphology-aware tokenization is employed, the perceived "difficulty" of these languages often diminishes.

The Curse of Multilinguality

Sharing parameters across 100+ languages leads to "negative transfer" or interference when model capacity is exceeded. Languages that are typologically distant from the high-resource "center" (typically English) suffer most from this gradient conflict and representation collapse.

Design Levers for Equitable NLP

The survey identifies several critical implications for future model design:

  • Normalization: Performance gaps shrink when models are normalized for sequence length, encoding efficiency, and semantic exposure. This suggests that "hard" languages are often just being measured with a biased yardstick.
  • Architectural Shifts: Moving toward byte- or character-level models can eliminate script-based unfairness.
  • Sampling Strategies: Implementing byte-normalized or information-normalized sampling ensures languages receive equivalent "semantic coverage" during training, regardless of raw data volume.
  • Evaluation Reform: Evaluation must move beyond subword-level perplexity toward typology-aware probes that test specific linguistic competencies.

Conclusion

Performance disparities in multilingual LMs are rarely due to the inherent difficulty of a language. Instead, they are primarily artifacts of engineering choices that favor English-like structures. By embracing linguistic diversity as a design constraint rather than an afterthought, the NLP community can build models that are truly global. Future work must move beyond low-level syntax to address higher-level challenges in pragmatics, discourse, and sociolinguistic variation.

Summary updated

Abstract

Multilingual language models (LMs) promise broader NLP access, yet current systems deliver uneven performance across the world's languages. This survey examines why these gaps persist and whether they reflect intrinsic linguistic difficulty or modeling artifacts. We organize the literature around two questions: do linguistic disparities arise from representation and allocation choices (e.g., tokenization, encoding, data exposure, parameter sharing) rather than inherent complexity; and which design choices mitigate inequities across typologically diverse languages. We review linguistic features, such as orthography, morphology, lexical diversity, syntax, information density, and typological distance, linking each to concrete modeling mechanisms. Gaps often shrink when segmentation, encoding, and data exposure are normalized, suggesting much apparent difficulty stems from current modeling choices. We synthesize these insights into design recommendations for tokenization, sampling, architectures, and evaluation to support more balanced multilingual LMs.
BibTeX
@misc{shani2026roots,
  author = {Chen Shani and Yuval Reif and Nathan Roll and Dan Jurafsky and Ekaterina Shutova},
  title = {{The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?}},
  year = {2026},
  eprint = {2601.07220},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.07220},
}
Figure