Introduction
The "pretrain and finetune" paradigm has long dominated natural language processing, yet it remains constrained by the requirement for vast amounts of labeled data for each specific task and language. Large language models (LLMs) like GPT-3 introduced a paradigm shift through "in-context learning," enabling models to perform tasks via zero-shot or few-shot prompting. However, these advancements have been largely English-centric, leaving a significant gap for low-resource and typologically diverse languages.
mGPT (Multilingual GPT) aims to bridge this digital divide by extending few-shot learning capabilities to 61 languages across 25 diverse language families. With a specific focus on underrepresented languages—including those of the CIS and indigenous peoples of Russia—mGPT demonstrates that large-scale autoregressive models can achieve competitive performance while fostering greater linguistic inclusivity.
Methodology
The researchers developed two model variants: a 1.3B parameter model (mGPT-XL) and a larger 13B parameter model. Both utilize a decoder-only Transformer architecture based on the GPT-3 design.
To handle the computational demands of a 13B parameter model, the architecture incorporates sparse attention, alternating between dense and sparse attention layers to efficiently capture long-range dependencies. A critical component of the multilingual setup was the tokenization strategy; after an ablation study, the authors selected Byte-level BPE (BBPE) with a 100,000-token vocabulary. This choice optimized perplexity across various scripts and ensured robust handling of diverse linguistic structures.
Training was conducted on 400 billion tokens. The 1.3B model was trained over 14 days using 256 GPUs, while the 13B model required 22 days on 512 GPUs.
Data Composition
The training corpus consisted of a 600GB blend of Wikipedia and the Colossal Clean Crawled Corpus (C4). While Wikipedia provided high-quality encyclopedic content, the C4 dataset was instrumental in bolstering the representation of "tail" languages—those with limited digital footprints.
As illustrated in the distribution above, the inclusion of mC4 significantly increased the token count for low-resource languages, which is essential for the model to learn the nuances of less-documented linguistic systems.
Performance and Insights
The models were evaluated across a comprehensive suite of benchmarks, including XGLUE, XTREME, XCOPA, XNLI, and mLAMA. mGPT-13B demonstrated performance comparable to XGLM-1.7B, despite supporting twice the number of languages. It showed particularly strong performance in the Austronesian, Japonic, and Romance language families.
The Script Effect and Multilinguality
Evaluation revealed several key linguistic insights. Interestingly, the model exhibited lower perplexity on non-Latin scripts (e.g., Cyrillic, Arabic). This "Script Effect" likely stems from the lower diversity of training data available for these languages (predominantly Wikipedia), making the text more predictable rather than the languages being inherently "easier."
Furthermore, the "Curse of Multilinguality" was observed: as the number of supported languages increases, the model's capacity per language is effectively diluted, leading to potential performance trade-offs. This was evident in sequence labeling tasks like NER, where the 1.3B model actually outperformed the 13B model in several scenarios, suggesting that smaller, denser representations may be more effective for certain structural linguistic tasks.
Few-Shot Dynamics and Knowledge Probing
mGPT exhibits the same prompt sensitivity as its English-only predecessors. In many cases, increasing the number of examples in a prompt (increasing $k$ in $k$-shot learning) did not yield linear improvements and sometimes led to degradation, suggesting a sensitivity to distribution shifts. In knowledge probing (mLAMA), the models achieved precision@1 scores above 0.6 for Germanic and Romance languages, though scores fell below 0.5 for Afro-Asiatic and Koreanic families.
Conclusion and Future Work
mGPT represents a significant step toward inclusive AI, proving that generative few-shot learners can be scaled to a wide array of languages. However, challenges remain regarding data quality for the lowest-resource languages and the inherent social biases present in web-scale corpora, particularly regarding gender and religion. Future research will focus on refined language-specific filtration, "prompt-tuning" to mitigate sensitivity, and safety-focused data to improve performance on sensitive tasks like hate speech detection, where the current model performs near random guessing.
Summary updated