When we train a large language model on a massive, multi-billion-token corpus spanning hundreds of languages, it is tempting to assume that the model develops a uniform, language-agnostic "intelligence." We expect that an optimization breakthrough or a prompting strategy that works in English will transfer seamlessly to French, Japanese, or Tamil.
However, empirical evidence reveals a highly jagged performance frontier. Multilingual models do not possess equal capabilities across their supported languages. Instead, they exhibit severe cross-lingual performance disparities. To understand why this happens, we must look beneath the surface of "multilingual" labels and examine how data asymmetries, structural typography, and alignment mechanics interact within a model's shared embedding space.
The Bottleneck of Tokenization and Vocabulary Allocation
The first reason models perform inconsistently across languages happens before text even reaches the transformer layers: tokenization. Most multilingual models use a shared vocabulary (e.g., Byte-Pair Encoding) trained on an aggregate corpus heavily skewed toward high-resource languages like English.
As a result, high-resource text is split into clean, morpheme-like tokens, while low-resource or morphologically rich languages are fractured into tiny, sub-word fragments or individual bytes. This fragmentation has severe consequences:
- It artificially inflates the sequence length for certain languages, exhausting the model's context window faster.
- It dilutes the semantic information carried by each token, forcing the model to allocate more parameters and attention steps just to reconstruct basic word meanings.
In her study on predicting human reading behavior, Ece Takmaz notes that these inherent variations in vocabulary size and corpus characteristics create significant performance disparities across languages (Team DMG at CMCL 2022 Shared Task: Transformer Adapters for the Multi- and Cross-Lingual Prediction of Human Reading Behavior). When a model's underlying vocabulary fails to represent a language's typological nuances efficiently, downstream performance suffers, regardless of how large the model is.
Typological and Script Complexity
Languages are not just different strings of text; they represent fundamentally different structural systems. A model optimized to process the relatively rigid word order and analytic structure of English often struggles when confronted with highly inflected, agglutinative, or non-alphabetic writing systems.
For instance, languages with rich morphology (such as Slavic or Dravidian languages) pack vast amounts of syntactic information—like case, gender, and tense—into complex word endings. If the training data is sparse, the model fails to generalize across these permutations. Even within the same language family, performance can vary wildly. Girma Yohannis Bade et al. observed this phenomenon when detecting AI-generated text, finding that BERT-based models achieved significantly higher effectiveness in Malayalam than in Tamil, despite both being low-resource Dravidian languages (Girma@DravidianLangTech 2025: Detecting AI Generated Product Reviews).
Furthermore, complex script systems introduce unique constraints. In developing the M-IFEval benchmark, Antoine Dussolle et al. demonstrated that a model's general instruction-following proficiency does not scale uniformly across linguistic boundaries (M-IFEval: Multilingual Instruction-Following Evaluation). Models that easily follow character-frequency constraints or script-exclusion rules in English frequently fail when executing the exact same constraints in languages with complex character boundaries, such as Japanese.
The Fragility of Multilingual In-Context Learning
A common assumption in modern NLP is that prompting techniques—such as few-shot in-context learning (ICL)—act as a universal lever to boost model performance. If a model struggles with a task, we simply provide a few examples in the prompt.
However, the mechanics of ICL are highly unstable across the multilingual spectrum. Miaoran Zhang et al. conducted a multidimensional analysis of demonstrations in multilingual ICL and discovered that the benefits are deeply inconsistent (The Impact of Demonstrations on Multilingual In-Context Learning: A Multidimensional Analysis). For state-of-the-art "chat" models, providing few-shot examples in non-English languages often yields no statistical advantage over a well-crafted zero-shot instruction template. The model's ability to map the latent pattern of a few examples into a correct execution strategy is highly dependent on the target language and task type, proving that prompting strategies do not generalize uniformly across model families.
Cultural Nuance and Representation Gaps
Finally, cross-lingual variation is driven by a data-representation gap. Language is inextricably linked to culture, meaning that certain tasks require localized contextual knowledge that cannot be acquired through literal translation.
Consider figurative language, such as euphemisms or offensive speech. A model might understand the literal definitions of words in a target language but fail to grasp their underlying connotations due to a lack of culturally relevant training data. Todd Firsich et al. highlighted this when testing GPT-4 on cross-lingual euphemism detection (Can GPT4 Detect Euphemisms across Multiple Languages?). They found that performance varied drastically across languages because advanced models face severe bottlenecks when training and testing data lack overlap in specific cultural and linguistic contexts. Without deep, localized data exposure, the model's semantic space for that language remains shallow, leading to sudden performance drops in tasks requiring nuanced comprehension.