TL;DR
This paper introduces DIPLomA, a novel method for efficiently adapting powerful instruction-following Large Language Models (LLMs) to low-resource languages. By decoupling language adaptation from instruction tuning and using a "delta merging" technique, DIPLomA significantly improves linguistic proficiency and instruction-following capabilities without requiring costly large-scale post-training on target-language instruction data.
What Problem Does This Paper Address?
Developing high-quality instruction-following LLMs, like those that can summarize text, answer questions, or follow complex commands, typically involves a multi-stage process. This includes extensive pre-training on massive datasets and then a costly post-training phase (instruction tuning and alignment) that requires large amounts of curated instruction data and human feedback. While this pipeline works well for high-resource languages like English, it creates a significant barrier for low-resource languages due to the scarcity and expense of obtaining such data. This paper aims to bridge this gap by enabling efficient adaptation of existing instruction-tuned LLMs to these underrepresented languages.
Methods
DIPLomA employs a two-stage approach:
- Continual Pre-training: A base pre-trained LLM is first adapted to a low-resource language by continuing its pre-training on monolingual data from that language. This step enhances the model's fluency and linguistic competence in the target language. To mitigate catastrophic forgetting of English capabilities, an anchor English corpus is mixed into this pre-training stage.
- Instruction Merging (Delta Merging): Instead of performing expensive post-training from scratch in the target language, DIPLomA leverages the "delta" – the weight differences between an instruction-tuned model and its base pre-trained counterpart. This delta, which encapsulates instruction-following and human preference alignment learned during large-scale English post-training, is merged into the language-adapted model using weight interpolation. This allows the transfer of these crucial capabilities without needing target-language instruction data.
Datasets
- Training Data:
- Basque: ZelaiHandi (531M words) for continual pre-training.
- Welsh & Swahili: FineWeb-2 (389M words for Welsh, 490M for Swahili) for continual pre-training.
- English: FineWeb (for mixed pre-training) and Magpie dataset (for mixed fine-tuning).
- Translated Instruction Data: Magpie-Llama-3.1-Pro-300K-Filtered dataset translated into Basque (MagpieEU, 261k pairs) and used in a mixed fashion (MagpieMIX) for fine-tuning baselines.
- Evaluation Benchmarks:
- Basque: NoRobotsEU (100 instructions) for instruction-following and linguistic proficiency. AdvBench (100 adversarial instructions) translated into Basque for safety.
- Welsh & Swahili: Translated versions of NoRobotsEU and AdvBench (using Google Translate) for instruction-following and safety.
- English: Evaluated using the same benchmarks as the original Llama-3.1-Instruct model to check for performance degradation.
Key Insights and Ideas
- Delta Merging for Low-Resource Adaptation: The core idea is that instruction-following capabilities are largely encoded in the difference (delta) between an instructed model and its base. This delta can be efficiently transferred, bypassing the need for expensive target-language instruction tuning.
- Decoupling Language Adaptation and Instruction Tuning: Separating these two stages allows for better specialization. First, build strong linguistic foundations in the target language, then inject instruction-following skills.
- Mitigating Catastrophic Forgetting: Mixing English data during the continual pre-training phase is crucial for preserving the model's general capabilities and performance in high-resource languages.
- Generalization of Safety Alignment: Safety alignment learned in English appears to transfer effectively to low-resource languages like Basque, suggesting that safety guardrails can be generalized.
- LLM-as-a-Judge Reliability: The study validates GPT-4o as a reliable proxy for human evaluation in low-resource settings, especially for instruction-following and safety, significantly improving scalability.
Key Results
- Basque: DIPLomA achieved substantial improvements, with a 54-point increase in instruction-following accuracy over the original Llama-3.1-Instruct and outperforming baselines by an average of 50 points. Linguistic proficiency also saw significant gains.
- Welsh & Swahili: DIPLomA demonstrated average relative improvements of 63 points in Welsh and 51 in Swahili for instruction-following, while maintaining strong English performance.
- English Performance: DIPLomA models retained high levels of instruction-following and safety in English, with some models even slightly improving. This contrasts with baselines that showed significant drops in English performance.
- Model Size and Architecture: The method proved effective across different LLM sizes (1B, 3B, 8B) and architectures (LLaMA, Gemma-2), showing scalability and robustness.
- Pre-training Strategy: A mixed pre-training strategy (80% target language, 20% English) was found to be crucial for mitigating catastrophic forgetting in English, while a Basque-only strategy led to a drastic drop in English performance.
Limitations and Future Work
- Model Access: The method requires access to both base and instruction-tuned versions of a model, limiting its applicability to open-source LLMs with publicly released variants.
- Scope of Evaluation: Human assessments were limited to Basque; Welsh and Swahili relied on GPT-4o. Evaluation benchmarks, though diverse, were limited in size.
- Future Work:
- Broader validation on more diverse model families and architectures.
- Exploring other LLM capabilities like multi-turn dialogue handling.
- Developing more comprehensive safety benchmarks and mitigation strategies tailored to low-resource languages.
- Investigating the impact of the merging coefficient ($\alpha$) further, noting a trade-off between instruction-following and linguistic proficiency/perplexity.
Summary updated