mGPT: Few-Shot Learners Go Multilingual

Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Anastasia Kozlova, Vladislav Mikhailov, Tatiana Shavrina

Introduction

The "pretrain and finetune" paradigm has long dominated natural language processing, yet it remains constrained by the requirement for vast amounts of labeled data for each specific task and language. Large language models (LLMs) like GPT-3 introduced a paradigm shift through "in-context learning," enabling models to perform tasks via zero-shot or few-shot prompting. However, these advancements have been largely English-centric, leaving a significant gap for low-resource and typologically diverse languages.

mGPT (Multilingual GPT) aims to bridge this digital divide by extending few-shot learning capabilities to 61 languages across 25 diverse language families. With a specific focus on underrepresented languages—including those of the CIS and indigenous peoples of Russia—mGPT demonstrates that large-scale autoregressive models can achieve competitive performance while fostering greater linguistic inclusivity.

Methodology

The researchers developed two model variants: a 1.3B parameter model (mGPT-XL) and a larger 13B parameter model. Both utilize a decoder-only Transformer architecture based on the GPT-3 design.

To handle the computational demands of a 13B parameter model, the architecture incorporates sparse attention, alternating between dense and sparse attention layers to efficiently capture long-range dependencies. A critical component of the multilingual setup was the tokenization strategy; after an ablation study, the authors selected Byte-level BPE (BBPE) with a 100,000-token vocabulary. This choice optimized perplexity across various scripts and ensured robust handling of diverse linguistic structures.

Training was conducted on 400 billion tokens. The 1.3B model was trained over 14 days using 256 GPUs, while the 13B model required 22 days on 512 GPUs.

Data Composition

The training corpus consisted of a 600GB blend of Wikipedia and the Colossal Clean Crawled Corpus (C4). While Wikipedia provided high-quality encyclopedic content, the C4 dataset was instrumental in bolstering the representation of "tail" languages—those with limited digital footprints.

Token distribution per language, highlighting the contribution of the mC4 dataset.

As illustrated in the distribution above, the inclusion of mC4 significantly increased the token count for low-resource languages, which is essential for the model to learn the nuances of less-documented linguistic systems.

Performance and Insights

The models were evaluated across a comprehensive suite of benchmarks, including XGLUE, XTREME, XCOPA, XNLI, and mLAMA. mGPT-13B demonstrated performance comparable to XGLM-1.7B, despite supporting twice the number of languages. It showed particularly strong performance in the Austronesian, Japonic, and Romance language families.

Perplexity comparison between mGPT-XL and mGPT-13B across 61 languages.

The Script Effect and Multilinguality

Evaluation revealed several key linguistic insights. Interestingly, the model exhibited lower perplexity on non-Latin scripts (e.g., Cyrillic, Arabic). This "Script Effect" likely stems from the lower diversity of training data available for these languages (predominantly Wikipedia), making the text more predictable rather than the languages being inherently "easier."

Furthermore, the "Curse of Multilinguality" was observed: as the number of supported languages increases, the model's capacity per language is effectively diluted, leading to potential performance trade-offs. This was evident in sequence labeling tasks like NER, where the 1.3B model actually outperformed the 13B model in several scenarios, suggesting that smaller, denser representations may be more effective for certain structural linguistic tasks.

Few-Shot Dynamics and Knowledge Probing

mGPT exhibits the same prompt sensitivity as its English-only predecessors. In many cases, increasing the number of examples in a prompt (increasing $k$ in $k$-shot learning) did not yield linear improvements and sometimes led to degradation, suggesting a sensitivity to distribution shifts. In knowledge probing (mLAMA), the models achieved precision@1 scores above 0.6 for Germanic and Romance languages, though scores fell below 0.5 for Afro-Asiatic and Koreanic families.

Conclusion and Future Work

mGPT represents a significant step toward inclusive AI, proving that generative few-shot learners can be scaled to a wide array of languages. However, challenges remain regarding data quality for the lowest-resource languages and the inherent social biases present in web-scale corpora, particularly regarding gender and religion. Future research will focus on refined language-specific filtration, "prompt-tuning" to mitigate sensitivity, and safety-focused data to improve performance on sensitive tasks like hate speech detection, where the current model performs near random guessing.

Summary updated

Abstract

This paper introduces mGPT, a multilingual variant of GPT-3, pretrained on 61 languages from 25 linguistically diverse language families using Wikipedia and the C4 Corpus. We detail the design and pretraining procedure. The models undergo an intrinsic and extrinsic evaluation: language modeling in all languages, downstream evaluation on cross-lingual NLU datasets and benchmarks in 33 languages, and world knowledge probing in 23 languages. The in-context learning abilities are on par with the contemporaneous language models while covering a larger number of languages, including underrepresented and low-resource languages of the Commonwealth of Independent States and the indigenous peoples in Russia. The source code and the language models are publicly available under the MIT license.
BibTeX
@article{shliazhko2024mgpt,
  author = {Oleh Shliazhko and Alena Fenogenova and Maria Tikhonova and Anastasia Kozlova and Vladislav Mikhailov and Tatiana Shavrina},
  title = {{mGPT: Few-Shot Learners Go Multilingual}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00633},
}
Figure