What Drives Performance in Multilingual Language Models?

Sina Bagheri Nezhad, Ameeta AgrawalNAACL 2024

TL;DR

This paper investigates what factors truly drive the performance of multilingual large language models (MLLMs) across a wide range of languages. They found that while pretraining data size is important for languages the model has seen, for unseen languages, script type and language family become critical, highlighting the importance of cross-lingual transfer. Interestingly, model size and architecture don't significantly alter these key drivers.

What Problem Does This Paper Address?

Multilingual Large Language Models (MLLMs) have revolutionized NLP, enabling applications across many languages. However, understanding why they perform well (or poorly) across such a diverse linguistic landscape remains a challenge. Previous research often focused on a limited set of languages or tasks, and crucially, didn't clearly distinguish between languages the model encountered during pretraining (SEEN) versus those it had never seen before (UNSEEN). This distinction is vital because the factors influencing performance might differ drastically when a model encounters a completely new language. This paper aims to bridge this gap by systematically analyzing the impact of various factors on MLLM performance across SEEN, UNSEEN, and ALL languages.

Methods

The researchers employed a comprehensive evaluation of 6 MLLMs, including Masked Language Models (MLMs), autoregressive models, and instruction-tuned LLMs. They used a text classification task on the SIB-200 dataset, which covers 204 languages. To understand the influence of different factors, they utilized decision tree analysis. This statistical method helps identify the relative importance of various features (like pretraining data size, language family, script type, and general resource availability) in predicting model performance (measured by F1 score). To validate the significance of the features identified by the decision trees, they also used the Mann-Whitney U test. They evaluated models in three scenarios: zero-shot, two-shot in-context learning (ICL), and full-shot (fine-tuning).

Datasets

The primary dataset used for evaluation was the SIB-200 dataset. This dataset is notable for its inclusion of 204 languages, making it a valuable resource for studying multilingual performance across a broad linguistic spectrum. The paper highlights that SIB-200 has a deliberate focus on low-resource languages, with most languages falling into resource level 1.

Key Insights and Ideas

  • Pretraining Data is Key, But Context Matters: For languages the MLLM has seen during pretraining (SEEN), the amount of pretraining data is a significant driver of performance. However, for languages the model has never seen (UNSEEN), pretraining data is irrelevant.
  • The Power of Linguistic Features for Unseen Languages: When faced with UNSEEN languages, MLLMs seem to rely heavily on script type and language family. This suggests that cross-lingual transfer learning, leveraging similarities in writing systems and linguistic relationships, is crucial for generalization.
  • Model Size Isn't Everything: Interestingly, the study found that model size and architecture did not significantly alter the most important factors identified by the decision trees. This implies that the fundamental linguistic characteristics of the data and the languages themselves play a more dominant role than the sheer scale of the model.
  • Resource Availability is Nuanced: While general resource availability (categorizing languages by how many speakers or resources they have) plays a role, it's less consistently influential than pretraining data (for SEEN) or linguistic features (for UNSEEN). It appears to be more important for specific models or settings.
  • Instruction Tuning's Influence: For instruction-tuned models like Bloomz, the initial pretraining corpus distribution seems to be more influential than the data used for instruction tuning itself, suggesting the foundational pretraining is critical.

Key Results

The decision tree analysis revealed distinct patterns of feature importance across the three scenarios (ALL, SEEN, UNSEEN):

  • ALL Languages: Pretraining data was the most influential factor in 29 out of 31 cases. This often boiled down to whether the language was encountered during training at all, rather than the specific quantity of data.
  • SEEN Languages: Pretraining data remained the most influential factor in 22 out of 31 combinations. Performance generally increased with more pretraining data. General resource availability also showed importance for some models (e.g., Bloom in zero-shot).
  • UNSEEN Languages: This is where the biggest shift occurred. Out of 31 models, 23 were most impacted by script type, and 8 by language family. This strongly supports the idea of cross-lingual transfer. For example, the Latin script was a key differentiator.

Limitations and Future Work

The authors acknowledge several limitations:

  • SIB-200 Dataset Biases: The SIB-200 dataset, while extensive, might have biases in language representation and genre distribution, potentially affecting the generalizability of findings.
  • Task Specificity: The analysis was conducted on a text classification task. Findings might not directly apply to other NLP tasks.
  • Model Landscape: The study analyzed a diverse set of models, but it may not be fully representative of the entire MLLM landscape.
  • Evolving Models: The findings are based on the current state of MLLMs, and relative factor importance could change as models evolve.

Future work suggested includes:

  • Investigating why script type can be more influential for specific models and settings.
  • Understanding why models lean towards different factors under different settings (e.g., zero-shot vs. two-shot).
  • Expanding the analysis to more diverse datasets, different NLP tasks, a broader range of models, and conducting longitudinal studies.

Summary updated

BibTeX
@inproceedings{nezhad2024what,
  author = {Sina Bagheri Nezhad and Ameeta Agrawal},
  title = {{What Drives Performance in Multilingual Language Models?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure