TL;DR
This paper overviews the IWPT 2020 Shared Task on Enhanced Universal Dependencies (EUD), a challenging task focused on parsing natural language into richer, more expressive dependency structures beyond basic UD. The task highlighted the progress and remaining challenges in EUD parsing, with participants achieving competitive results despite the increased complexity of the annotation scheme.
What Problem Does This Paper Address?
Traditional Universal Dependencies (UD) provide a consistent framework for cross-linguistic syntactic annotation, but they primarily represent basic dependency trees. However, certain linguistic phenomena, like ellipsis (gapping) and complex coordination structures, are difficult to capture with simple, single-parent edges. Enhanced Universal Dependencies (EUD) address this by allowing for richer graph structures, including multi-parent nodes and "empty" nodes to represent missing elements. The problem this paper tackles is evaluating the state-of-the-art in parsing natural language into these more complex EUD structures, understanding the challenges involved, and assessing the performance of various approaches.
Methods
The shared task involved participants developing systems to parse raw text into EUD. The evaluation was based on datasets covering 17 languages. The paper discusses various approaches taken by participants, broadly categorized into two main strategies:
- Conversion-based: Systems first parse into standard UD and then apply hand-written rules or heuristics to convert these basic trees into EUD. This approach leverages existing robust UD parsers.
- Direct graph-based: Systems are designed to directly produce EUD graphs, often using graph-based parsing techniques that can naturally handle cycles and multi-parent structures.
The paper also details the creation of the EUD datasets, which were based on UD release 2.5 but enriched with additional enhancements for the task. Simple heuristics were applied to fill in missing enhancements where possible, aiming for uniformity.
Datasets
The evaluation was conducted on datasets for 17 languages from four language families: Arabic, Bulgarian, Czech, Dutch, English, Estonian, Finnish, French, Italian, Latvian, Lithuanian, Polish, Russian, Slovak, Swedish, Tamil, and Ukrainian. These datasets were derived from the Universal Dependencies (UD) release 2.5, with some treebanks having richer enhanced annotations than others. For the shared task, treebanks for the same language were merged to promote robust parsers.
Key Insights and Ideas
- EUD is a significant step up in complexity: The enhanced layer allows for a much richer representation of linguistic phenomena, but this comes with increased parsing difficulty.
- Heuristics can bridge annotation gaps: The authors successfully used simple heuristics to add missing enhancements to some treebanks, demonstrating a practical way to create more uniform datasets for shared tasks.
- Evaluation challenges: The inherent differences in enhancement levels across treebanks necessitated "coarse" and "qualitative" evaluation metrics to ensure fair comparison, as systems might be penalized for predicting enhancements not present in the gold standard.
- Empty nodes are tricky: Handling "empty" nodes, crucial for phenomena like gapping, presents challenges in terms of their exact placement and how to represent their information, leading to a proposed solution of expanding dependency labels to include paths.
- Minimal performance drop: Despite the increased complexity, the performance drop from basic UD parsing to EUD parsing was minimal for most systems, suggesting that current parsing techniques are adaptable.
Key Results
The shared task saw competitive results, with the top systems achieving ELAS (Enhanced Labeled Attachment Score) scores comparable to LAS (Labeled Attachment Score) for basic UD parsing in previous multilingual tasks. Table 3 shows that the best systems achieved ELAS scores in the high 80s for some languages. The paper highlights that while the overall accuracy is high, the difference between EULAS (ignoring subtypes) and ELAS (including subtypes) suggests that predicting the correct label enhancements is still a challenge for some approaches. Qualitative evaluation revealed that the impact of enhancements varied significantly across languages, with some languages showing a higher percentage of "new" or "structurally new" dependencies.
Limitations and Future Work
The paper acknowledges several limitations:
- Inconsistent annotation: Not all UD treebanks include the same level of enhancements, making it harder to train a single, multilingual model and posing challenges for evaluation. While UD release 2.6 improved this, it remains an issue.
- Validation constraints: The strict validation rules for EUD graphs (e.g., connectivity, single root) were a surprise to some participants and required post-hoc correction scripts.
- Empty node representation: The paper notes that their solution for empty nodes, while functional, might lose information about sibling relationships among empty nodes.
Future work suggested includes:
- Investigating alternative representations for empty nodes or different evaluation strategies to better handle them.
- More clearly defining validation constraints for future EUD tasks.
- Further research into how to best leverage the richer information provided by EUD for downstream applications.
Summary updated