Large Language Models (LLMs) demonstrate remarkable capabilities in reasoning, coding, and creative writing. However, these capabilities are not distributed equally across the world's languages. Modern LLMs exhibit a profound performance disparity, excelling in English while faltering on non-English and low-resource languages.
To understand why this gap exists and how to bridge it, we must look beneath the surface of multilingual benchmarks and examine how data imbalances shape the internal representations of neural networks.
The Root of the Disparity: Parameter Optimization and Data Imbalance
The primary driver of the cross-lingual performance gap is the composition of pre-training corpora. The vast majority of web-scale data used to train foundational models is English-centric. As a result, the model's internal parameter space is overwhelmingly optimized to capture the syntax, semantics, and logical structures of English.
When a model is trained predominantly on one language, its attention heads and feed-forward networks form dense, highly efficient pathways for processing that language. As observed by Christoph Leiter et al. in ChatGPT: A Meta-Analysis after 2.5 Months, this data imbalance directly correlates with user satisfaction and perceived utility across different global communities, creating a "linguistic optimism bias" where English users experience a much more capable tool than non-English users.
For underrepresented languages, the model's internal representations are sparse and poorly optimized. This structural deficiency manifests in several ways:
- Higher Hallucination Rates: In language education contexts, Zilin Wang et al. note that data imbalances lead to significantly higher rates of hallucinations and lower content quality for learners of minority or ancient languages (A Systematic Review of Generative Artificial Intelligence in Language Education).
- The Fine-Tuning Paradox: Simply fine-tuning an English-centric model on low-resource data does not easily fix the problem. Andre Freitas et al. discovered that adapting open-source models to Italian using low-quality or translated instruction data can actually degrade the model's underlying reasoning capabilities, revealing a tension between linguistic proficiency and logical robustness (Teasing LLMs Adapted to Italian).
Unlocking Latent Knowledge via Cross-Lingual Pivoting
If an LLM's core reasoning capabilities are deeply tied to its English parameter pathways, how can we leverage those capabilities for non-English queries? One powerful intuition is to use English as an internal "thinking pivot."
Even when a model struggles to generate a direct response in a low-resource language, it often retains a latent, cross-lingual mapping of concepts. By structuring the prompt to force the model to process logic in English before translating the final output, researchers can bypass the weaker, non-English parameter pathways.
[User Query in Target Language]
│
▼
[Step 1: Translate Query to English Pivot]
│
▼
[Step 2: Perform Logical Reasoning / Chain-of-Thought in English]
│
▼
[Step 3: Translate Reasoning & Answer back to Target Language]
│
▼
[Final Output to User]
This mechanism underpins Cross-Lingual-Thought (XLT) prompting, introduced by Haoyang Huang et al. in Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting. By routing arithmetic and logical tasks through an English logical template, the authors unlocked massive performance boosts across 27 languages.
Similarly, the Pivot Language Guided generation (PLUG) framework, proposed by Zhihan Zhang, demonstrates that instructing a model to draft its response in a high-resource pivot language before responding in the target language yields a 29% average improvement in instruction-following for low-resource languages (PLUG: Leveraging Pivot Language in Cross-Lingual Instruction Tuning).
Aligning Semantic Spaces
While prompting strategies exploit existing pathways, a more permanent solution is to explicitly align the semantic spaces of different languages during alignment or instruction-tuning.
When an LLM processes different languages, it maps tokens into a high-dimensional vector space. If the training data is imbalanced, the vector space for English will be highly structured and nuanced, while the space for a low-resource language will be isolated and fragmented. Cross-lingual alignment techniques aim to project the isolated language vectors directly onto the well-optimized English vectors.
Wenhao Zhu et al. achieved this using Cross-lingual Instruction-Tuning (CoIT) (Extrapolating Large Language Models to Non-English by Aligning Languages). By mixing parallel translation tasks with translated general instructions, CoIT forces the model to close the geometric distance between language representations in its hidden layers. This alignment allows the model to transfer its foundational reasoning capabilities from English to languages with entirely different scripts or grammatical structures without the prohibitive computational cost of full pre-training.
The Path to Native Multilingual Models
While pivoting and semantic alignment mitigate the symptoms of English-centric bias, achieving true equity across languages requires building large-scale models natively tailored to non-English structures.
To bypass the limitations of English-centric foundations, researchers are increasingly developing native, large-scale architectures trained on diverse regional corpora. For example, El Moatez Billah Nagoudi et al. introduced the JASMINE suite to establish state-of-the-art benchmarks for Arabic few-shot learning (JASMINE: Arabic GPT Models for Few-Shot Learning), while Xu Han et al. developed CPM to address structural characteristics unique to Chinese text (CPM: A Large-scale Generative Chinese Pre-trained Language Model).
By combining native pre-training with rigorous, localized evaluation frameworks—such as the KazMMLU benchmark introduced by Mukhammed Togmanov et al. (KazMMLU: Evaluating Language Models on Kazakh, Russian, and Regional Knowledge of Kazakhstan)—the NLP community is moving away from a singular, English-centric view of artificial intelligence toward a more balanced, multilingual future.