TL;DR
This paper introduces "active forgetting" as a pretraining strategy for decoder-only Large Language Models (LLMs) to improve their ability to transfer knowledge to new languages. By strategically "forgetting" certain learned representations during pretraining, these models demonstrate enhanced cross-lingual transfer capabilities, leading to better performance on downstream tasks in unseen languages without sacrificing performance on previously learned ones.
What Problem Does This Paper Address?
Large Language Models (LLMs) excel in English but often struggle with multilingual tasks. While encoder-only models have shown impressive cross-lingual transfer (performing well on new languages even when trained only on English), decoder-only LLMs, which are crucial for generative tasks, have lagged in this regard. Traditional methods for adapting LLMs to new languages, such as vocabulary expansion and fine-tuning, often lead to performance degradation on existing languages and don't fully leverage the potential for cross-lingual transfer. This paper aims to bridge this gap by proposing a pretraining method that specifically enhances the cross-lingual transfer capabilities of decoder-only LLMs.
Methods
The core innovation is the application of "active forgetting" during the pretraining phase of decoder-only LLMs. Inspired by prior work on encoder-only models, this strategy involves periodically resetting token embeddings to random values during pretraining. This forces the model to relearn representations, promoting greater "plasticity" and adaptability.
The standard adaptation process for LLMs to new languages typically involves:
- Vocabulary Expansion: Learning a new vocabulary for the target languages and merging it with the existing vocabulary.
- Fine-tuning: Adapting the language modeling head and new token embeddings using standard language modeling on the new language data, while keeping the base model frozen.
- Instruction Fine-tuning: Further fine-tuning the adapted model on English instruction data.
The authors hypothesize that if the base LLM is pretrained using active forgetting, the subsequent adaptation and instruction fine-tuning will result in significantly better cross-lingual transfer.
Datasets
- Pretraining Data: Wikipedia dumps of 12 languages were used for the initial pretraining of the base LLMs.
- Adaptation Data: Wikipedia dumps of 14 new languages, disjoint from the pretraining set, were used for adapting the models.
- English Instruction Tuning Data: OpenOrca dataset was used for instruction fine-tuning.
- Evaluation Benchmarks:
- 6 multilingual benchmarks (details not fully specified but follow Aggarwal et al. (2024)).
- Perplexity and isotropy evaluation on 50 languages (26 "other" languages not in pretraining or adapting sets).
- 4-shot translation (English-to-X) performance on the same 50 languages using the FLORES-200 dataset.
- Specific tasks evaluated include belebele, MLQA, XCOPA, XLSUM, and XQUAD.
Key Insights and Ideas
- Vocabulary Expansion Can Hurt: The common method of adapting LLMs via vocabulary expansion can improve performance on new languages but often at the expense of performance on previously learned languages. This suggests that simply adding new vocabulary isn't a universally beneficial strategy.
- Active Forgetting Enhances Plasticity: Pretraining with active forgetting leads to more adaptable and "plastic" multilingual representations, allowing models to learn new languages more effectively.
- Improved Multilingual Representations: Models pretrained with active forgetting exhibit better quality multilingual representations, as evidenced by lower perplexity and better isotropy scores.
- Consistent Gains Across Model Sizes: The benefits of active forgetting pretraining were observed consistently across different model scales (400M, 782M, 1.6B, and 2.8B parameters).
Key Results
- Cross-Lingual Transfer: Active forgetting (AFA) models outperformed baseline models on 6 out of 7 multilingual benchmarks.
- Perplexity and Isotropy: AFA models consistently achieved lower perplexity and better isotropy scores compared to both baseline and baseline-adapted models, indicating higher quality multilingual representations.
- Translation Performance: AFA models showed significant improvements in translation tasks, especially for "adapting" languages, without the degradation seen in baseline-adapted models on "other" languages. For example, in Table 3, AFA models often show better overall translation scores than baseline-adapted models.
- Task Performance: Across various tasks (belebele, MLQA, XCOPA, XLSUM, XQUAD), AFA models generally outperformed baseline and baseline-adapted models, particularly in the "adapting" and "overall" categories. For instance, in Table 2, AFA models show lower (better) isotropy scores across different parameter sizes.
Limitations and Future Work
- Direct Application: The proposed method is for pretraining new LLMs. It cannot be directly applied to existing, already trained LLMs.
- Simulating Active Forgetting: The authors suggest exploring methods to simulate active forgetting on intermediate checkpoints of open-source LLMs.
- Scale of Experiments: The current experiments were conducted on models that could fit within their compute budget, and the data size was smaller than that used for models like XLM-R. Further evaluation on larger-scale models and datasets is needed.
- Adaptation Methods: Future work could explore more effective language adaptation methods for existing finetuned LLMs.
- Ethics: The authors emphasize the need for thorough studies of intrinsic and extrinsic biases before deploying LLMs whose pretraining directly affects token embeddings.
Summary updated