Kardeş-NLU: Transfer to Low-Resource Languages with the Help of a High-Resource Cousin – A Benchmark and Evaluation for Turkic Languages

Lütfi Kerem Senel, Benedikt Ebing, Konul Baghirova, Hinrich Schuetze, Goran Glavašš2024 · DOI

Abstract

Lütfi Kerem Senel, Benedikt Ebing, Konul Baghirova, Hinrich Schuetze, Goran Glavaš. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

2024.

Introduction

Massively multilingual language models (mmLMs) such as XLM-R have significantly advanced natural language processing. However, these models often underperform on low-resource (LR) languages that are underrepresented in their pretraining corpora—a phenomenon known as the "curse of multilinguality." While most cross-lingual transfer (XLT) research focuses on transferring knowledge from English to LR languages, many LR languages possess a high-resource (HR) "cousin" language with shared grammatical structures and vocabulary.

This paper introduces Kardeş-NLU, a benchmark for five low-resource Turkic languages, and investigates whether Turkish, a high-resource relative, can serve as a more effective bridge for knowledge transfer than English alone.

The Kardeş-NLU Benchmark

The authors developed Kardeş-NLU to evaluate Natural Language Understanding (NLU) across five Turkic languages: Azerbaijani, Kazakh, Kyrgyz, Uzbek, and Uyghur. The benchmark consists of high-quality, human-post-edited translations of three standard tasks:

  • Natural Language Inference (NLI): Based on XNLI.
  • Semantic Textual Similarity (STS): Based on STS-B.
  • Choice of Plausible Alternatives (COPA): Based on XCOPA.

Methodology

The study evaluates strategies to bridge the gap between English and LR Turkic languages through two main phases:

Intermediate Language Modeling (MLM)

Before task-specific fine-tuning, the mmLM (XLM-R) undergoes continued training on raw text using three approaches:

  1. TLLM: Training exclusively on the target LR language.
  2. BALM (Bilingual Alternating): Alternating batches between the LR language and Turkish.
  3. BJLM (Bilingual Joint): Mixing LR and Turkish data within the same batch to provide a stable representational anchor and prevent overfitting to small LR datasets.

Downstream Transfer

The researchers experimented with zero-shot and few-shot transfer using data from English, machine-translated Turkish, or a combination of both. Additionally, they benchmarked instruction-tuned Large Language Models (LLMs) like mT0 and ChatGPT in zero-shot settings.

Key Results and Insights

The experimental results demonstrate the significant advantage of leveraging a linguistic "cousin."

The "Cousin" Advantage and BJLM

Jointly training on the LR language and Turkish (BJLM) consistently outperformed training on the LR language alone. This strategy, combined with joint English and Turkish fine-tuning, yielded a 6.6% improvement in NLI and a 2.1% improvement in COPA over the vanilla XLM-R baseline.

Detailed performance metrics for NLI, COPA, and STS across different transfer and intermediate training settings.

LLM Performance vs. Fine-Tuning

The comparison between state-of-the-art LLMs and fine-tuned models revealed notable inconsistencies. While mT0-XXL achieved a superior average accuracy of 71.1% on the COPA task, it lagged significantly on NLI, scoring only 58.2% compared to the 73.3% achieved by the fine-tuned BJLM model. This suggests that massive scale and instruction tuning do not universally solve low-resource challenges.

Comparison of mT0, ChatGPT, and the best XLT models on NLI and COPA tasks across five Turkic languages.

The Impact of Script

The results for Uyghur highlighted the importance of intermediate training for script adaptation. Although XLM-R was pretrained on Uyghur in Arabic script, the benchmark uses Cyrillic. Intermediate training on Cyrillic text led to dramatic performance gains, such as an increase from 42.2% to 59.4% in NLI, highlighting how intermediate training can resolve "unseen script" issues.

Conclusion

Kardeş-NLU demonstrates that using a high-resource "cousin" language like Turkish significantly enhances cross-lingual transfer for low-resource Turkic languages. The success of the BJLM approach suggests that linguistic proximity provides a powerful anchor for model representations. Future work may extend this strategy to other language families, such as using Spanish as a bridge for indigenous languages of the Americas.

Summary updated

BibTeX
@misc{senel2024kardes,
  author = {Lütfi Kerem Senel and Benedikt Ebing and Konul Baghirova and Hinrich Schuetze and Goran Glavašš},
  title = {{Kardeş-NLU: Transfer to Low-Resource Languages with the Help of a High-Resource Cousin – A Benchmark and Evaluation for Turkic Languages}},
  year = {2024},
  doi = {10.18653/v1/2024.eacl-long.100},
}
Figure