TL;DR
This paper introduces LLAMMAS, the first open-source instruction-following Large Language Model (LLM) for Estonian, a low-resource language. By combining cross-lingual instruction tuning with additional monolingual pretraining on Llama 2, the authors demonstrate effective knowledge transfer from English to Estonian, significantly improving performance on various tasks.
What Problem Does This Paper Address?
Large Language Models (LLMs) have shown remarkable capabilities, but their development and widespread adoption are heavily skewed towards high-resource languages like English. This leaves many languages, such as Estonian, underserved. Adapting existing LLMs to these low-resource languages is challenging due to the lack of extensive training data and foundation models. Instruction tuning, a method to align LLMs with human preferences, is primarily English-centric. This paper tackles the problem of creating instruction-following LLMs for low-resource languages by exploring cost-efficient methods for cross-lingual knowledge transfer.
Methods
The core of this work involves adapting the Llama 2 model for Estonian. The authors investigate two primary strategies:
- Monolingual Continued Pretraining: This involves further pretraining the Llama 2 model on a mix of Estonian and English data. The goal is to make the model more familiar with Estonian without forgetting its English capabilities.
- Cross-Lingual Instruction Tuning: This technique involves training the model on instructions in both English and Estonian. The paper explores different combinations of general task instructions and translation task instructions. They also introduce "Alpaca-est," a novel, general-task instruction dataset for Estonian, and "HQI" (High-Quality Instructions) and "HQTRTASK" (High-Quality Translation Task Instructions) datasets derived from high-quality English sources.
The authors also experiment with sequential training (translation first, then general tasks) versus combined datasets for instruction tuning.
Datasets
- Alpaca-cleaned: A cleaned version of the original Stanford Alpaca dataset.
- Alpaca-est: A newly generated Estonian version of the Alpaca dataset, containing 52,006 instructions.
- HQI (High-Quality Instructions): A dataset of high-quality English instructions, including Open Assistant 1 multi-turn conversations and examples from Chain-of-Thought and FLAN-2.
- TRTASK: Translation task instructions generated from relatively low-quality parallel bitexts (CCMatrix, WikiMatrix, OpenSubtitles, Europarl), with a 75% English-to-Estonian and 25% Estonian-to-English split.
- HQTRTASK: High-quality parallel data from WMT18 dev set and MTee held-out validation dataset, converted into instruction format.
- CulturaX: Used for pretraining data, a subset of Estonian and English data.
Evaluation datasets include EstQA, EstCOPA, EstGEC-L2, W&I+LOCNESS, FLORES-200, MMLU, TruthfulQA, WinoGrande, TriviaQA, and HellaSwag.
Key Insights and Ideas
- Cross-lingual knowledge transfer is effective: Even a small amount of additional monolingual pretraining followed by cross-lingual instruction tuning significantly boosts performance on Estonian tasks.
- Monolingual pretraining diminishes the need for translation tasks: When sufficient monolingual pretraining is performed, the specific benefit of translation-task instructions during fine-tuning decreases.
- High-quality English instructions transfer well: Supplementing general task instructions with high-quality English instructions (HQI) leads to positive cross-lingual knowledge transfer into Estonian, improving commonsense reasoning and conversational abilities.
- Sequential training can be beneficial: For certain tasks, training sequentially (e.g., translation tasks first, then general tasks) can yield better results than combining all instructions into a single dataset.
- Estonian LLMs can be conversational: LLAMMAS demonstrates the ability to engage in multi-turn conversations in Estonian, despite not being trained on Estonian conversational data.
Key Results
- The best model, LLAMMAS, achieved competitive zero-shot performance on multiple Estonian tasks.
- LLAMMAS outperforms the baseline Llama-2-7B on EstQA (84% vs. 60.3% F1) and EstCOPA (83% vs. 51.8% accuracy).
- For machine translation (ET→EN), LLAMMAS-MT achieved a BLEU score of 36.9, competitive with GPT-3.5-turbo.
- Grammatical error correction (GEC) also saw significant improvements.
- While LLAMMAS-BASE showed a slight performance drop on some English benchmarks (average 1.2%), it still performed competitively.
Limitations and Future Work
- Dependence on proprietary LLM data: The Alpaca-est dataset was generated using OpenAI's proprietary LLMs, which may lead to imitation of style rather than factuality.
- Limited Estonian benchmarks: The evaluation is constrained by the availability of benchmarks for Estonian NLP tasks.
- Safety evaluation: The released model lacks comprehensive safety evaluation and is intended for research purposes.
- Bias in self-instruct data: Instructions generated via self-instruct were not manually checked, potentially introducing risks like bias.
Future work should focus on evaluating the harmlessness and helpfulness of LLMs for Estonian, as this has not been done for proprietary LLMs supporting the language. Further research into more robust safety evaluations and expanding the range of Estonian benchmarks is also needed.
Summary updated