TL;DR
This paper introduces the first comprehensive set of open-source foundation models and benchmarks for the Azerbaijani language. By releasing a large new corpus (DOLLMA), a family of BERT-class models (aLLMA), and several evaluation datasets, the authors aim to democratize access to advanced NLP capabilities for this low-resource language, addressing the current reliance on proprietary, cloud-based solutions.
What Problem Does This Paper Address?
Large Language Models (LLMs) have revolutionized Natural Language Processing (NLP), but their benefits have not been evenly distributed across languages. Low-resource languages, like Azerbaijani (spoken by 24 million people), often lag behind due to a lack of readily available, high-quality open-source models and evaluation benchmarks. While proprietary models like GPT-4 offer some support, they are costly and inaccessible for many. Existing open-source multilingual models show limited proficiency in Azerbaijani, and attempts to create Azerbaijani-specific models have been hampered by a lack of systematic benchmarking and community adoption. This paper aims to bridge this gap by providing the foundational resources needed to advance Azerbaijani NLP research and development.
Methods
The core of this work involves creating and leveraging several key components:
- DOLLMA Corpus: A new, large text corpus specifically curated for Azerbaijani, containing approximately 651.1 million words. This corpus was built by combining various sources, including Wikipedia, news articles, books, and legal texts, with careful consideration for data quality and linguistic variety.
- aLLMA Models: A family of encoder-only foundation models (aLLMA-Small, aLLMA-Base, aLLMA-Large) trained from scratch on the DOLLMA corpus. These models are inspired by the BERT architecture and are designed to be adaptable for various downstream NLP tasks. Dynamic masking was employed during pre-training to improve robustness.
- New Benchmarks: To facilitate evaluation, the authors introduce three new Azerbaijani-specific datasets:
- AZE-SCI: For text classification.
- AZE-NSP: For next-sentence prediction.
-
CB-MCQ: A challenging closed-book question-answering dataset. These are complemented by existing translated datasets like WikiANN, SQuAD, LDQuAd, and MRPC.
-
Comprehensive Evaluation: The paper presents a benchmark that evaluates aLLMA models alongside various multilingual models (e.g., XLM-ROBERTa, mDeBERTa-v3, BERT-Base-MULTI) and further pre-trained Azerbaijani models. The evaluation uses consistent hyperparameters to ensure fair comparison.
Datasets
- Training Data:
- DOLLMA: A newly created corpus of 651.1 million words for Azerbaijani, comprising:
- English Wikipedia (translated) * Azerbaijani Wikipedia * News articles * Books (two subsets, Books I and Books II) * Blogs * Azerbaijani laws
- Evaluation Datasets:
- Newly Introduced:
- AZE-SCI (Text Classification)
- AZE-NSP (Next-Sentence Prediction)
- CB-MCQ (Closed-Book Question-Answering)
- Existing (Translated/Adapted):
- WikiANN (Named Entity Recognition)
- SQuAD (Extractive QA)
- LDQuAd (Extractive QA)
- MRPC (Paraphrase Identification)
- Newly Introduced:
Key Insights and Ideas
- Pre-training from scratch is viable: The aLLMA models demonstrate that training foundation models from scratch on a dedicated low-resource language corpus can yield competitive performance, even against larger, multilingual models.
- Multilingual models offer surprising robustness: Even models that were not specifically trained or fine-tuned for Azerbaijani (like XLM-ROBERTa and mDeBERTa-v3) show strong performance on several tasks, highlighting the transfer learning capabilities of well-trained multilingual models.
- Further pre-training can help, but with caveats: While further pre-training multilingual models on Azerbaijani data (e.g., BERT-Base-AZE) can improve performance, there's a risk of "model collapse" on certain tasks, as seen with LDQuAd.
- English-only models can be useful baselines: The original English-only BERT-Base model performed surprisingly well on some tasks, suggesting that even these models can be leveraged for Azerbaijani NLP with appropriate fine-tuning.
- The need for specialized benchmarks: The introduction of AZE-SCI, AZE-NSP, and CB-MCQ addresses a critical gap in evaluating Azerbaijani NLP models, moving beyond simple classification to more complex understanding tasks.
Key Results
- The aLLMA-Base model, trained from scratch, achieved the best overall performance among models of similar size in 4 out of 6 benchmark tasks.
- mDeBERTa-v3 and aLLMA-Base showed the best overall performance across all tested models.
- BERT-Scratch performed particularly well on AZE-SCI, MRPC, and WikiANN, suggesting its effectiveness for tasks solvable with statistical information.
- Multilingual models like XLM-ROBERTa-Large and mDeBERTa-v3 consistently outperformed many other models, even though Azerbaijani was not a primary focus.
- The CB-MCQ benchmark proved to be a significant challenge, with all tested models failing to learn it effectively, indicating an open research problem.
Limitations and Future Work
- Model Size and Corpus Size: The authors acknowledge that time and hardware limitations restricted the size of the aLLMA models and the extent of pre-training. Training larger models on larger corpora is a clear path for improvement.
- Domain Distribution: The DOLLMA corpus has an imbalanced domain distribution, lacking sufficient first-person narrative and STEM fields. Future versions aim to improve this.
- Deduplication: Deduplication was not performed on the initial DOLLMA corpus, which is planned for future iterations to allow for the incorporation of potentially overlapping text sources.
- Benchmarks: While new benchmarks were introduced, the authors emphasize the need for more labeled task-specific datasets and the inclusion of other existing models in the leaderboards.
- Generative Models: This paper focused on encoder-only models. Developing generative foundation models for Azerbaijani is identified as an ambitious but important future direction.
- Next-Sentence Prediction in Pre-training: Including NSP in the pre-training phase for aLLMA models could further boost performance.
Summary updated