Turkish Treebanking: Unifying and Constructing Efforts

Utku Türk, Furkan Atmaca, Şaziye Betül Özateş, Abdullatif Köksal, Balkiz Ozturk Basaran, Tunga Gungor, Arzucan ÖzgürACL 2019

TL;DR

This paper addresses the fragmentation and inconsistency in existing Turkish treebanks, crucial for advancing Turkish Natural Language Processing (NLP). The authors present a re-annotated Turkish PUD treebank and the initial construction of the Turkish National Corpus UD (TNC-UD) treebank, aiming to unify annotation styles and improve parsing accuracy.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) heavily relies on treebanks – annotated datasets that represent the syntactic structure of sentences. However, for many languages, including Turkish, there's a significant imbalance in the availability and quality of these resources. Existing Turkish treebanks suffer from inconsistent annotation schemes, a lack of detailed documentation on the annotation process, and often rely on automatic conversion from non-UD (Universal Dependencies) styles, leading to errors and a fragmented landscape. This inconsistency hinders the development of robust Turkish NLP tools and applications. The authors aim to bridge this gap by creating more unified, linguistically consistent, and well-documented Turkish treebanks.

Methods

The core of this work involves two main efforts:

  1. Re-annotation of the Turkish PUD Treebank: The authors manually re-annotated the existing Turkish PUD treebank to ensure consistency with Universal Dependencies (UD) guidelines and a more refined interpretation of the Stanford Dependencies (SD) scheme. This involved simplifying syntactic relation tags and addressing specific linguistic phenomena like compounds and light verb constructions.
  2. Construction of the TNC-UD Treebank: This involved annotating a new corpus, the Turkish National Corpus (TNC), following the same unified UD and SD guidelines. The initial release includes 500 sentences, with plans to expand to 10,000. Morphological analyses were automatically generated using a deep learning-based parser trained on a re-annotated Turkish IMST-UD treebank.

A significant methodological contribution is the development of a custom, open-source desktop annotation tool. This tool features a tabular view, hierarchical tree visualization, and advanced filtering and morphological editing capabilities, designed to streamline the annotation process, especially for agglutinative languages like Turkish.

Datasets

  • Turkish PUD Treebank: Re-annotated version.
  • Turkish National Corpus (TNC): A new corpus used for the TNC-UD treebank, initially comprising 500 sentences from 5 registers (essays, newspapers, instructional texts, popular culture, biographical texts), with plans to expand to 10,000 sentences.
  • Turkish IMST-UD Treebank: Used for training the morphological analyzer.

Key Insights and Ideas

  • The Importance of Unified Annotation: The paper strongly emphasizes that a consistent annotation scheme across different Turkish treebanks is crucial for building reliable NLP tools. Inconsistencies, even minor ones, can significantly impact parsing accuracy.
  • Linguistic Nuances in Turkish: The re-annotation process highlighted specific challenges in Turkish, such as the complex nature of compounds and the appropriate tagging of light verb constructions, which required careful linguistic decisions.
  • Custom Tooling for Specific Needs: The development of a specialized annotation tool underscores the need for tailored software solutions that cater to the unique characteristics of languages like Turkish, improving annotator efficiency and reducing errors.
  • The Trade-off Between Simplicity and Accuracy: The results suggest that more linguistically precise annotations, while potentially harder for models to learn with limited data, lead to better long-term parsing performance and a more accurate representation of language structure.

Key Results

  • Improved Parsing Accuracy: While the re-annotated Turkish PUD treebank (BPUD) showed a slight decrease in attachment scores compared to the original, this was attributed to the more rigorous annotation. When TNC-UD data was added, parsing accuracy for the PUD treebank improved.
  • High Inter-Annotator Agreement: The manual annotation process achieved high inter-annotator agreement (99% for heads, 88% for dependency labels), indicating the effectiveness of the developed guidelines and the annotators' expertise.
  • Availability of Resources: The re-annotated PUD treebank, the TNC-UD treebank (initial version), annotation guidelines, and the custom annotation software are publicly available under an open license.

Limitations and Future Work

  • Limited TNC-UD Size: The current TNC-UD treebank is relatively small (500 sentences), which limits its immediate impact on parsing performance. The authors plan to expand it to 10,000 sentences.
  • Automatic vs. Manual Annotation: While morphological analysis was automated, syntactic analysis relied on manual annotation. The paper acknowledges the need for further work to refine these processes.
  • Future Work: The authors plan to complete the annotation of the TNC-UD to 10,000 sentences. Future enhancements could include human-validated morphological analyses, prosodic information, and detailed semantic analysis. They also aim to continue re-annotating the IMST-UD treebank.

Summary updated

BibTeX
@inproceedings{turk2019turkish,
  author = {Utku Türk and Furkan Atmaca and Şaziye Betül Özateş and Abdullatif Köksal and Balkiz Ozturk Basaran and Tunga Gungor and Arzucan Özgür},
  title = {{Turkish Treebanking: Unifying and Constructing Efforts}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure