The Hebrew Universal Dependency Treebank: Past Present and Future

Shoval Sade, Amit Seker, Reut TsarfatyEMNLP 2018

TL;DR

This paper addresses the challenge of creating a consistent and high-quality Universal Dependencies (UD) treebank for Hebrew by manually validating and correcting the existing UDv2 version. The authors demonstrate that improving linguistic coherence and annotation consistency leads to better parsing performance, and they propose future directions for standardizing UD across languages and integrating it with external linguistic resources.

What Problem Does This Paper Address?

Developing robust natural language processing (NLP) tools for any language relies heavily on high-quality annotated data, often in the form of treebanks. For Modern Hebrew, the Hebrew Treebank (HTB) has been the primary resource for training and evaluating parsers for two decades. However, the HTB has undergone numerous automatic and semi-automatic conversions over the years, leading to inconsistencies and deviations from the Universal Dependencies (UD) guidelines. This lack of standardization makes it difficult to compare results across different studies and hinders the development of reliable NLP applications. This paper tackles the problem of harmonizing the Hebrew UD treebank with the broader UD framework, aiming to improve its linguistic accuracy and parsing performance.

Methods

The core of this paper involves a manual validation and revision process of the existing UDv2 Hebrew treebank (UD HTB). The authors systematically reviewed the UDv2 HTB, identifying and correcting errors related to:

  • Argument Type Classification: Ensuring consistency with UD guidelines.
  • Morphological vs. Syntactic Property Mix-ups: Clarifying the distinction between features and relations.
  • Language-Specific vs. Universal Phenomena: Reducing the reliance on language-specific sub-typing where universal labels exist.

Based on these corrections, they created a new version called UDv2New. They then used both UDv2 and UDv2New to train a morphosyntactic parser and evaluated its performance in both ideal (gold morphology) and realistic (predicted morphology) scenarios.

Datasets

  • Hebrew Treebank (HTB): The primary dataset, consisting of 6221 morpho-syntactically annotated newspaper sentences. The paper focuses on the UDv2 version and introduces a revised version, UDv2New.
  • UDv2 HTB: The existing Universal Dependencies version of the HTB, used as a baseline for comparison.
  • UDv2New HTB: The newly created, manually validated and corrected version of the UD HTB.

Key Insights and Ideas

  • Manual Validation is Crucial: Automatic conversions, while efficient, can introduce subtle but significant errors that require manual linguistic expertise to identify and rectify.
  • Coherence Drives Performance: Improving the linguistic coherence and annotation consistency of a treebank directly translates to better parsing performance, even with the same underlying parser.
  • UD as a Unifying Force: The UD initiative is a significant step towards standardizing linguistic resources and metrics, benefiting not only cross-linguistic research but also individual language NLP communities.
  • Interface Challenges: A key challenge lies in standardizing the interface between UD treebanks and external morphological/lexical resources, especially for morphologically rich languages.
  • Language-Specific vs. Universal Representation: There's a need for clearer UD guidelines on how to distinguish and represent language-specific phenomena versus universal ones, particularly concerning subtypes and features.

Key Results

  • Improved Parsing Performance: The UDv2New version of the HTB led to improved parsing performance compared to the UDv2 version in ideal scenarios (gold morphology). Specifically, LAS (Labeled Attachment Score), MLAS (Mean Labeled Attachment Score), and BLEX (Best Lexicalized Attachment Score) all saw improvements.
  • Realistic Scenario Challenges: In realistic scenarios (predicted morphology), the performance drop was substantial. However, UDv2New showed no significant improvement over UDv2 in this setting, suggesting issues with lexical coverage or mismatches between the revised scheme and external lexicons.
  • Infused Scenario Validation: When the correct morphological analysis was injected (infused scenario), UDv2New showed substantial improvements, indicating that the drop in realistic scenarios was indeed due to insufficient lexical coverage or mismatches.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that some inconsistencies remain unfixed due to the complexity of their automatic retrieval and correction. Examples include the representation of "it-extraposition" and the handling of dative case nuances.
  • In realistic parsing scenarios, the performance gains were limited, highlighting the dependency on external lexical resources and the need for better harmonization.
  • Future Work:
    • A comprehensive revision of lexical and morphological resources associated with the UD scheme is planned to improve empirical parsing results in realistic scenarios.
  • The authors aim to add an "enhanced dependencies" layer to the treebank, which would be more suitable for downstream semantic tasks.
  • They propose a future UD treebank scheme revision that focuses on the universal organization of grammar, with specific attention to subtypes, features, lexical resources, and semantic applications.

Summary updated

BibTeX
@inproceedings{sade2018hebrew,
  author = {Shoval Sade and Amit Seker and Reut Tsarfaty},
  title = {{The Hebrew Universal Dependency Treebank: Past Present and Future}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure