mGPT: Few-Shot Learners Go Multilingual

Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, Tatiana Shavrina

Introduction

Large language models (LLMs) have revolutionized natural language processing, yet they remain heavily biased toward English. This linguistic imbalance leaves a significant gap in support for the world's 7,000+ languages, particularly for minority and low-resource groups. While "pretrain and finetune" models like mBERT exist, they often require substantial labeled data—a resource that is non-existent for many languages.

To bridge this "digital divide," researchers developed mGPT, a multilingual adaptation of the GPT-3 architecture. Pretrained on 61 diverse languages across 25 language families, mGPT demonstrates that massively multilingual generative models can achieve competitive zero-shot and few-shot performance through in-context learning, even on typologically distant or under-represented languages.

Methodology

The researchers developed two versions of mGPT: a 1.3B parameter model (mGPT-XL) and a larger 13B parameter model (mGPT-13B).

  • Architecture: The models utilize a Transformer decoder architecture with alternating dense and sparse attention layers, mirroring the GPT-3 design to handle long-range dependencies efficiently.
  • Tokenization: Representing diverse scripts is a core challenge. After an ablation study of five strategies, the team settled on Byte-level BPE (BBPE) with a 100k vocabulary size, which provided the most robust perplexity results across different scripts.
  • Training: The models were trained using DeepSpeed and Megatron-LM on V100 GPU clusters, processing approximately 400 billion tokens over 600,000 training steps.

Data and Language Coverage

The training corpus combined filtered data from Wikipedia and the Colossal Clean Crawled Corpus (C4). A primary goal was the inclusion of languages from the Commonwealth of Independent States (CIS) and indigenous groups of Russia that are typically absent from LLM research.

Table 1: Distribution of the 61 languages across 25 language families included in mGPT training.

Key Results and Scaling

The evaluation spanned NLU benchmarks (XNLI, PAWS-X), knowledge probing (mLAMA), and sequence labeling. The results indicate that mGPT-1.3B performs on par with larger competitors like XGLM-1.7B while covering twice as many languages.

Figure 1: Average perplexity scores across language families for mGPT-XL (1.3B) and mGPT-13B models.

Scaling and the "Curse of Multilinguality"

Scaling from 1.3B to 13B parameters significantly boosted performance for high-resource languages, but the researchers observed diminishing returns for low-resource languages. This suggests that raw parameter count is not the only bottleneck for minority language performance. Furthermore, increasing language diversity can lead to performance degradation if model capacity is not balanced with careful corpus weighting—a phenomenon known as the "curse of multilinguality."

Discussion and Limitations

While mGPT represents a major step forward, several challenges remain:

  • Evaluation Gap: There is a severe lack of high-quality, human-verified benchmarks for the very low-resource languages the model targets.
  • Prompt Sensitivity: Performance is highly sensitive to prompt phrasing. In some complex tasks, adding more examples (few-shot) actually caused performance to drop compared to zero-shot.
  • Social Risks: The model struggled with hate speech detection, highlighting that representation biases in training data remain a significant risk.

Future work will focus on multilingual-specific scaling laws and techniques like prompt tuning to improve the stability and robustness of these models across the global linguistic landscape.

Summary updated

BibTeX
@article{shliazhko2024mgpt,
  author = {Oleh Shliazhko and Alena Fenogenova and Maria Tikhonova and Vladislav Mikhailov and Anastasia Kozlova and Tatiana Shavrina},
  title = {{mGPT: Few-Shot Learners Go Multilingual}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00633},
}
Figure