Abstract
Large-scale generative language models like GPT-3 have demonstrated remarkable few-shot learning capabilities but remain heavily biased toward English, with non-English data comprising only a small fraction of their training sets. This paper introduces XGLM, a family of multilingual generative models (up to 7.5 billion parameters) designed to extend the "in-context learning" paradigm to a global scale. By training on a balanced corpus of 30 diverse languages, XGLM achieves state-of-the-art performance in multilingual reasoning and translation, demonstrating that cross-lingual prompting can effectively unlock capabilities in low-resource languages.
Addressing the Multilingual Gap
While models like GPT-3 are famous for their ability to perform new tasks via few-shot prompting, their utility is limited for the majority of the world's population. Approximately 93% of GPT-3's training data is English, leaving other languages severely underrepresented. Existing multilingual models, such as mBERT or XLM-R, are typically encoder-only architectures that require extensive fine-tuning on labeled data for specific tasks. XGLM addresses this by providing a decoder-only generative model capable of zero-shot and few-shot tasks across many languages without parameter updates.
Methodology and Data Balancing
The XGLM family includes four versions, ranging from 564 million to 7.5 billion parameters. The architecture follows a standard decoder-only Transformer design but incorporates a significantly larger joint vocabulary of 250,000 sub-word tokens to support diverse scripts.
To mitigate the dominance of English in web-crawled data, the researchers employed a multinomial up-sampling strategy. This ensures that low-resource languages, such as Swahili or Telugu, are represented more frequently during training than their natural occurrence in the CC100-XL dataset would dictate.
Evaluation and Benchmarking
The models were evaluated on a suite of multilingual benchmarks, including XNLI (inference), XCOPA (commonsense reasoning), and PAWS-X (paraphrasing). The authors also introduced XStoryCloze, a professionally translated version of the StoryCloze dataset covering 10 typologically diverse languages.
Researchers explored three prompting strategies:
- Native prompts: Handcrafted in the target language.
- Translated prompts: English prompts translated into the target language.
- Cross-lingual prompting: Using English instructions for non-English data.
Key Results and Insights
XGLM 7.5B demonstrated significant improvements over similarly sized English-centric models. It outperformed GPT-3 on multilingual commonsense reasoning by 7.4% and on natural language inference by 5.4% in zero-shot settings. In machine translation (FLORES-101), XGLM surpassed GPT-3 in 171 out of 182 directions.
One of the most striking findings was the effectiveness of the "Universal" English Prompt. Using English templates often yielded better results for non-English tasks than native-language prompts. This suggests that the model learns task logic primarily from its high-volume English data and can transfer that logic to other languages. Furthermore, the benefits of few-shot learning scaled significantly with model size, with the 7.5B model leveraging examples much more effectively than smaller variants.
Limitations and Future Directions
Despite its success, XGLM faces a "curse of multilinguality": teaching the model 30 languages results in a roughly 10% performance drop on pure English tasks compared to an English-only model of the same size. Additionally, while XGLM retains factual knowledge better across languages, it still struggles with few-shot hate speech detection, highlighting the need for more diverse safety evaluation datasets. Future work will focus on scaling the model further and refining data filtering to improve quality.
Summary updated