Abstract
Lütfi Kerem Senel, Benedikt Ebing, Konul Baghirova, Hinrich Schuetze, Goran Glavaš. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).
2024.
Introduction
Massively multilingual language models (mmLMs) such as XLM-R have significantly advanced natural language processing. However, these models often underperform on low-resource (LR) languages that are underrepresented in their pretraining corpora—a phenomenon known as the "curse of multilinguality." While most cross-lingual transfer (XLT) research focuses on transferring knowledge from English to LR languages, many LR languages possess a high-resource (HR) "cousin" language with shared grammatical structures and vocabulary.
This paper introduces Kardeş-NLU, a benchmark for five low-resource Turkic languages, and investigates whether Turkish, a high-resource relative, can serve as a more effective bridge for knowledge transfer than English alone.
The Kardeş-NLU Benchmark
The authors developed Kardeş-NLU to evaluate Natural Language Understanding (NLU) across five Turkic languages: Azerbaijani, Kazakh, Kyrgyz, Uzbek, and Uyghur. The benchmark consists of high-quality, human-post-edited translations of three standard tasks:
- Natural Language Inference (NLI): Based on XNLI.
- Semantic Textual Similarity (STS): Based on STS-B.
- Choice of Plausible Alternatives (COPA): Based on XCOPA.
Methodology
The study evaluates strategies to bridge the gap between English and LR Turkic languages through two main phases:
Intermediate Language Modeling (MLM)
Before task-specific fine-tuning, the mmLM (XLM-R) undergoes continued training on raw text using three approaches:
- TLLM: Training exclusively on the target LR language.
- BALM (Bilingual Alternating): Alternating batches between the LR language and Turkish.
- BJLM (Bilingual Joint): Mixing LR and Turkish data within the same batch to provide a stable representational anchor and prevent overfitting to small LR datasets.
Downstream Transfer
The researchers experimented with zero-shot and few-shot transfer using data from English, machine-translated Turkish, or a combination of both. Additionally, they benchmarked instruction-tuned Large Language Models (LLMs) like mT0 and ChatGPT in zero-shot settings.
Key Results and Insights
The experimental results demonstrate the significant advantage of leveraging a linguistic "cousin."
The "Cousin" Advantage and BJLM
Jointly training on the LR language and Turkish (BJLM) consistently outperformed training on the LR language alone. This strategy, combined with joint English and Turkish fine-tuning, yielded a 6.6% improvement in NLI and a 2.1% improvement in COPA over the vanilla XLM-R baseline.
LLM Performance vs. Fine-Tuning
The comparison between state-of-the-art LLMs and fine-tuned models revealed notable inconsistencies. While mT0-XXL achieved a superior average accuracy of 71.1% on the COPA task, it lagged significantly on NLI, scoring only 58.2% compared to the 73.3% achieved by the fine-tuned BJLM model. This suggests that massive scale and instruction tuning do not universally solve low-resource challenges.
The Impact of Script
The results for Uyghur highlighted the importance of intermediate training for script adaptation. Although XLM-R was pretrained on Uyghur in Arabic script, the benchmark uses Cyrillic. Intermediate training on Cyrillic text led to dramatic performance gains, such as an increase from 42.2% to 59.4% in NLI, highlighting how intermediate training can resolve "unseen script" issues.
Conclusion
Kardeş-NLU demonstrates that using a high-resource "cousin" language like Turkish significantly enhances cross-lingual transfer for low-resource Turkic languages. The success of the BJLM approach suggests that linguistic proximity provides a powerful anchor for model representations. Future work may extend this strategy to other language families, such as using Spanish as a bridge for indigenous languages of the Americas.
Summary updated