Mukayese: Turkish NLP Strikes Back

Ali Safaya, Emirhan Kurtulus, Arda Göktogan, Deniz YüretACL 2022 · DOI

Introduction

Turkish is often categorized as an "under-researched" language. While it possesses millions of speakers and a vast digital footprint, it has historically lacked the organized benchmarks, standardized datasets, and public leaderboards necessary to drive systematic NLP progress. Without these tools, researchers often utilize disparate dataset versions or non-comparable metrics, making it difficult to identify true state-of-the-art (SOTA) performance.

MUKAYESE (Turkish for "comparison") addresses this gap by providing a comprehensive benchmarking suite. By introducing four new datasets and establishing standardized baselines across seven core tasks, MUKAYESE serves as a "north star" for the Turkish research community, demonstrating that even languages with complex linguistic structures can achieve high-level results with the right infrastructure.

The MUKAYESE Framework

The researchers developed a rigorous evaluation pipeline covering seven diverse NLP tasks: Language Modeling, Machine Translation, Named-Entity Recognition (NER), Sentence Segmentation, Spellchecking & Correction, Summarization, and Text Classification. For each task, they provide at least two baseline models, ranging from traditional rule-based systems to modern Transformers.

Overview of the MUKAYESE benchmarking suite, including tasks, datasets, metrics, and baseline models.

A primary contribution of MUKAYESE is the creation of standardized train/validation/test splits for existing datasets and the introduction of four brand-new datasets:

  • TRNEWS-64 and TRWIKI-67 for Language Modeling.
  • TRSEG-41 for Sentence Segmentation (curated from scientific abstracts, news, and tweets).
  • TRSPELL-10 for Spellchecking (10K words with artificial and human-verified misspellings).

Linguistic Challenges and Insights

Turkish presents unique challenges due to its agglutinative morphology. A single word can encapsulate the meaning of an entire English phrase through the addition of various suffixes, making tasks like spellchecking and language modeling particularly difficult compared to analytic languages like English.

Morphological breakdown of the Turkish word "arabanızdaydı" ('it was in your car'), illustrating the language's agglutinative nature.

This complexity leads to several key insights from the benchmarking process:

  • Architecture Matters: In Language Modeling, the SHA-RNN (a Recurrent Neural Network variant) outperformed the Adaptive Transformer on Turkish corpora. This suggests that for certain languages or data scales, RNN-based architectures still hold significant value.
  • Robustness Gaps: Sentence segmentation performance drops significantly when moving from "clean" editorial text to "corrupted" social media text, highlighting a major area where current models fail to generalize.
  • The Power of Pre-training: Across nearly all tasks, pre-trained multilingual models (like mBART50) or Turkish-specific models (like BERTurk) significantly outperformed models trained from scratch.

Key Results

MUKAYESE establishes high-performance baselines across its task suite, setting a new standard for Turkish NLP:

  • NER: BERTurk-CRF achieved a new SOTA on the Milliyet-NER dataset with a 96.48 F1 score, outperforming previous benchmarks by over 3 points.
  • Spellchecking: A new Hunspell-based dictionary reached a 99.62 F1 score for error detection, significantly surpassing the widely-used Zemberek tool.
  • Summarization: Using MT5-Base, the researchers achieved a ROUGE-L score of 39.92 on the MLSUM dataset.
  • Translation: Pre-trained mBART50 achieved a BLEU score of 24.17 on WMT-16 (tr-en), a major improvement over models trained from scratch (17.29).

Conclusion

MUKAYESE provides the necessary infrastructure to catalyze Turkish NLP research. While it covers seven major tasks, the authors acknowledge that future work is needed in areas like dependency parsing and in-context spellchecking. They hope this framework serves as a template for other under-researched language communities to build their own standardized benchmarking ecosystems.

Summary updated

BibTeX
@inproceedings{safaya2022mukayese,
  author = {Ali Safaya and Emirhan Kurtulus and Arda Göktogan and Deniz Yüret},
  title = {{Mukayese: Turkish NLP Strikes Back}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.69},
}
Figure