From Raw Text to Enhanced Universal Dependencies: The Parsing Shared Task at IWPT 2021

Gosse Bouma, Djamé Seddah, Daniel ZemanACL 2021

TL;DR

This paper details the second IWPT shared task focused on parsing raw text into Enhanced Universal Dependencies (EUD), a richer linguistic representation than standard dependency trees. The task saw significant improvements in performance, with systems achieving higher accuracy and better handling of complex linguistic phenomena, pushing the boundaries of automated natural language understanding.

What Problem Does This Paper Address?

Traditional dependency parsing, while powerful, often struggles to capture the full complexity of natural language. Phenomena like ellipsis (missing words), coordination (joining phrases), and relative clauses can be difficult to represent accurately with simple tree structures where each word has only one parent. Enhanced Universal Dependencies (EUD) aims to address this by introducing a more flexible graph-based representation that allows for multiple parents, "empty" nodes to represent elided material, and richer edge labels. This richer annotation is crucial for downstream applications that require a deeper semantic understanding of text. The IWPT 2021 shared task was designed to drive progress in automatically generating these EUD graphs from raw text, building upon the success and lessons learned from its 2020 predecessor.

Methods

The core of the task involved participants developing systems to parse raw text into EUD. Most systems employed a graph-based approach, often using biaffine functions to predict parent-child relationships between tokens. A common strategy was to first generate a basic dependency tree and then enhance it, or to directly produce a graph. Many systems leveraged pre-trained language models, with XML-R being a popular choice. To handle the complexity of EUD labels, particularly those involving case information, systems often de-lexicalized these labels during training, replacing them with simpler representations that could be reconstructed post-processing. The evaluation script itself also employed a strategy of collapsing "empty" nodes and reflecting their information in complex dependency labels.

Datasets

The evaluation was conducted on datasets spanning 17 languages from four language families. These datasets were based on the Universal Dependencies (UD) release 2.7. For several treebanks, the enhanced annotation was richer than what was officially released. Notably, the English data included the GUM treebank, and some French treebanks featured more conservative annotations compared to previous work. The task also involved harmonizing annotations across different treebanks within the same language to ensure a more uniform evaluation.

Key Insights and Ideas

  • EUD is Crucial for Deeper Understanding: The paper reinforces that EUD is essential for capturing linguistic phenomena that basic dependency trees cannot adequately represent, paving the way for more sophisticated NLP applications.
  • Graph-Based Parsing is Dominant: Unlike previous efforts that might convert EUD to trees, this year's task saw a clear shift towards graph-based parsing approaches, reflecting the inherent nature of EUD.
  • De-lexicalization is Key for Robustness: Simplifying complex, case-enhanced dependency labels into more general forms for training, and then reconstructing them, proved to be a successful strategy for improving system performance.
  • Uniformity Drives Progress: Harmonizing annotations across different treebanks within a language was a critical step that likely contributed to the overall performance improvements observed in the task.
  • Handling Empty Nodes is Tricky: While gapping and ellipsis are important phenomena, accurately representing the information conveyed by "empty" nodes remains a challenge, often requiring creative solutions in both annotation and evaluation.

Key Results

The shared task demonstrated significant progress in EUD parsing. The best performing system achieved an ELAS (labeled attachment score on enhanced dependencies) of 89.24%, a notable increase from the previous year's best score of 84.50%. The average ELAS for the top-5 systems also rose from 78.75% to 86.14%. Improvements were observed across most languages, with 11 out of 17 languages seeing ELAS scores above 90%. The paper also provides detailed ELAS scores per phenomenon, highlighting that while systems generally perform well, there's still variation in their ability to handle specific linguistic structures like coordination and relative clauses.

Limitations and Future Work

The authors acknowledge that some treebanks still have less uniform annotations, and that the task's evaluation script has limitations in handling multiple empty nodes in a sentence. While the overall performance has improved, there's still room for growth, particularly in languages with limited data. Future work could focus on further harmonizing annotations, developing more robust methods for handling empty nodes, and exploring more advanced graph-based parsing techniques. The increased performance suggests that EUD parsing is becoming more mature, but there's still a significant gap between automated systems and human annotators for certain complex phenomena.

Summary updated

BibTeX
@inproceedings{bouma2021from,
  author = {Gosse Bouma and Djamé Seddah and Daniel Zeman},
  title = {{From Raw Text to Enhanced Universal Dependencies: The Parsing Shared Task at IWPT 2021}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure