Efficient Large Scale Language Modeling with Mixtures of Experts

Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeffrey Wang, Luke Zettlemoyer, Mona T. Diab, Zornitsa Kozareva, Veselin StoyanovEMNLP 2022 · OpenReview

Abstract

Large-scale generative language models like GPT-3 have demonstrated remarkable few-shot learning capabilities but remain heavily biased toward English, with non-English data comprising only a small fraction of their training sets. This paper introduces XGLM, a family of multilingual generative models (up to 7.5 billion parameters) designed to extend the "in-context learning" paradigm to a global scale. By training on a balanced corpus of 30 diverse languages, XGLM achieves state-of-the-art performance in multilingual reasoning and translation, demonstrating that cross-lingual prompting can effectively unlock capabilities in low-resource languages.

Addressing the Multilingual Gap

While models like GPT-3 are famous for their ability to perform new tasks via few-shot prompting, their utility is limited for the majority of the world's population. Approximately 93% of GPT-3's training data is English, leaving other languages severely underrepresented. Existing multilingual models, such as mBERT or XLM-R, are typically encoder-only architectures that require extensive fine-tuning on labeled data for specific tasks. XGLM addresses this by providing a decoder-only generative model capable of zero-shot and few-shot tasks across many languages without parameter updates.

Methodology and Data Balancing

The XGLM family includes four versions, ranging from 564 million to 7.5 billion parameters. The architecture follows a standard decoder-only Transformer design but incorporates a significantly larger joint vocabulary of 250,000 sub-word tokens to support diverse scripts.

To mitigate the dominance of English in web-crawled data, the researchers employed a multinomial up-sampling strategy. This ensures that low-resource languages, such as Swahili or Telugu, are represented more frequently during training than their natural occurrence in the CC100-XL dataset would dictate.

Language distribution comparison between XGLM (pre- and post-sharding) and GPT-3, highlighting the more balanced representation in XGLM.

Evaluation and Benchmarking

The models were evaluated on a suite of multilingual benchmarks, including XNLI (inference), XCOPA (commonsense reasoning), and PAWS-X (paraphrasing). The authors also introduced XStoryCloze, a professionally translated version of the StoryCloze dataset covering 10 typologically diverse languages.

Researchers explored three prompting strategies:

  1. Native prompts: Handcrafted in the target language.
  2. Translated prompts: English prompts translated into the target language.
  3. Cross-lingual prompting: Using English instructions for non-English data.

Key Results and Insights

XGLM 7.5B demonstrated significant improvements over similarly sized English-centric models. It outperformed GPT-3 on multilingual commonsense reasoning by 7.4% and on natural language inference by 5.4% in zero-shot settings. In machine translation (FLORES-101), XGLM surpassed GPT-3 in 171 out of 182 directions.

Accuracy curves for StoryCloze, COPA, Winogrande, and HellaSwag benchmarks showing performance gains as the number of few-shot samples increases.

One of the most striking findings was the effectiveness of the "Universal" English Prompt. Using English templates often yielded better results for non-English tasks than native-language prompts. This suggests that the model learns task logic primarily from its high-volume English data and can transfer that logic to other languages. Furthermore, the benefits of few-shot learning scaled significantly with model size, with the 7.5B model leveraging examples much more effectively than smaller variants.

Limitations and Future Directions

Despite its success, XGLM faces a "curse of multilinguality": teaching the model 30 languages results in a roughly 10% performance drop on pure English tasks compared to an English-only model of the same size. Additionally, while XGLM retains factual knowledge better across languages, it still struggles with few-shot hate speech detection, highlighting the need for more diverse safety evaluation datasets. Future work will focus on scaling the model further and refining data filtering to improve quality.

Summary updated

BibTeX
@inproceedings{artetxe2022efficient,
  author = {Mikel Artetxe and Shruti Bhosale and Naman Goyal and Todor Mihaylov and Myle Ott and Sam Shleifer and Xi Victoria Lin and Jingfei Du and Srinivasan Iyer and Ramakanth Pasunuru and Giridharan Anantharaman and Xian Li and Shuohui Chen and Halil Akin and Mandeep Baines and Louis Martin and Xing Zhou and Punit Singh Koura and Brian O'Horo and Jeffrey Wang and Luke Zettlemoyer and Mona T. Diab and Zornitsa Kozareva and Veselin Stoyanov},
  title = {{Efficient Large Scale Language Modeling with Mixtures of Experts}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=UJ2oaJ69JaO},
}
Figure