Uncovering inequalities in new knowledge learning by large language models across different languages

Chenglong Wang, Haoyu Tang, Xiyuan Yang, Yueqi Xie, Jina Suh, Sunayana Sitaram, Junming Huang, Zhaoya Gong, Xing Xie, Fangzhao Wu

Abstract

As Large Language Models (LLMs) are increasingly integrated into global digital infrastructure, their ability to acquire new information becomes a critical measure of utility and equity. This paper investigates the "dynamic" inequality present in how LLMs learn new knowledge across high-resource and low-resource languages. By evaluating models like GPT-4o-mini and Llama-3.1-8B across 17 languages, we identify a significant performance gap. High-resource languages (HRLs) exhibit faster learning, superior cross-lingual transfer, and greater resistance to misinformation compared to low-resource languages (LRLs). These findings suggest that current AI architectures reinforce linguistic hegemony, potentially widening the global digital divide.

Introduction

Most research into multilingual Large Language Models (LLMs) focuses on "static" evaluation—measuring the knowledge a model acquired during its initial pre-training phase. However, in a rapidly evolving world, the true utility of an AI lies in its capacity for "dynamic" learning: the ability to incorporate new information through In-Context Learning (ICL) or fine-tuning.

This study addresses a critical question: Does the efficiency of new knowledge acquisition vary across languages? If LLMs learn more slowly or inaccurately in certain languages, users of those languages face a systematic disadvantage. We explore this "learning inequality" across four dimensions: effectiveness, transferability, prioritization, and robustness.

Methodology

Models and Languages

The researchers evaluated two state-of-the-art models: the proprietary GPT-4o-mini and the open-source Llama-3.1-8B. The study spanned 17 languages, categorized into 10 high-resource languages (HRLs) and 7 low-resource languages (LRLs). Languages were classified as LRL if they accounted for less than 0.1% of the GPT-3 pre-training corpus.

Learning Settings

Two primary methods of knowledge acquisition were tested:

  1. Fine-tuning: Updating model weights using LoRA (for Llama) or official APIs (for GPT).
  2. In-Context Learning (ICL): Providing new information directly within the prompt context.

Datasets

To prevent data leakage and ensure the models were learning truly "new" information, two novel parallel datasets were constructed:

  • Multilingual Parallel Fictional New Knowledge Dataset: 100 question-answer pairs regarding a hypothetical future (e.g., health tracking via genetic analytics in 2048).
  • Multilingual Parallel Common-sense Dataset: 50 pairs of basic facts (e.g., water freezing into ice) paired with deliberate misinformation (e.g., water freezing into steam) to test robustness.

Results and Analysis

1. The Efficiency Gap

High-resource languages demonstrate a clear "pre-optimization" advantage. In fine-tuning experiments, GPT-4o-mini reached 80% accuracy in HRLs after only 3 epochs. In contrast, it required 8 epochs to reach the same threshold in LRLs. Furthermore, while HRLs eventually plateaued at over 90% accuracy, LRLs struggled to surpass 80%, indicating a lower ceiling for knowledge retention in underrepresented languages.

2. Knowledge Transferability

The study found that knowledge is often "trapped" within LRLs. When GPT-4o-mini was fine-tuned on English data, it achieved 97% accuracy in English and transferred that knowledge effectively to other HRLs. However, when queried in Zulu, accuracy plummeted to 19%. This suggests that while models can generalize across dominant languages, they fail to bridge the gap to low-resource ones.

3. Linguistic Hegemony and Conflict

When models were presented with conflicting information—where one fact was provided in English and a contradictory fact in an LRL—the models exhibited a strong bias toward the HRL. Regardless of the language used for the final query, the model's response aligned with the high-resource version approximately 70% of the time. This positions HRLs as the "authoritative" source within the model's internal logic.

4. Robustness to Misinformation

LRLs proved significantly more fragile when exposed to incorrect data. In Zulu, GPT-4o-mini’s accuracy dropped from 80% to 40% upon the introduction of a single piece of misinformation. HRLs maintained much higher stability, suggesting that the lack of linguistic data makes LRLs more susceptible to hallucinations and AI-generated errors.

Discussion

The findings suggest that the digital divide is being reinforced through the very mechanisms used to update AI models. The "Efficiency Gap" implies that the internal architectures of LLMs are fundamentally better at processing the structures of HRLs. This creates a cycle where dominant languages become more useful and reliable, while low-resource languages remain prone to error and slower to adapt to new information.

Conclusion

This research highlights a systemic inequality in how LLMs learn. To achieve global AI equity, the community must move beyond static benchmarks and address these dynamic learning disparities. Future work should focus on cross-disciplinary strategies, including multilingual instruction tuning and more balanced continual pre-training, to ensure that the benefits of AI are accessible regardless of a user's native language.

Summary updated

Abstract

As large language models (LLMs) gradually become integral tools for problem solving in daily life worldwide, understanding linguistic inequality is becoming increasingly important. Existing research has primarily focused on static analyses that assess the disparities in the existing knowledge and capabilities of LLMs across languages. However, LLMs are continuously evolving, acquiring new knowledge to generate up-to-date, domain-specific responses. Investigating linguistic inequalities within this dynamic process is, therefore, also essential. In this paper, we explore inequalities in new knowledge learning by LLMs across different languages and four key dimensions: effectiveness, transferability, prioritization, and robustness. Through extensive experiments under two settings (in-context learning and fine-tuning) using both proprietary and open-source models, we demonstrate that low-resource languages consistently face disadvantages across all four dimensions. By shedding light on these disparities, we aim to raise awareness of linguistic inequalities in LLMs' new knowledge learning, fostering the development of more inclusive and equitable future LLMs.
BibTeX
@inproceedings{wang2025uncovering,
  author = {Chenglong Wang and Haoyu Tang and Xiyuan Yang and Yueqi Xie and Jina Suh and Sunayana Sitaram and Junming Huang and Yu Xie and Zhaoya Gong and Xing Xie and Fangzhao Wu},
  title = {{Uncovering inequalities in new knowledge learning by large language models across different languages}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2025},
  doi = {10.1073/pnas.2514626122},
}
Figure