Enhanced Universal Dependency Parsing with Second-Order Inference and Mixture of Training Data

Xinyu Wang, Yong Jiang, Kewei Tu

Abstract

This paper presents the system used in our submission to the IWPT 2020 Shared Task. Our system is a graph-based parser with second-order inference. For the low-resource Tamil corpus, we specially mixed the training data of Tamil with other languages and significantly improved the performance of Tamil. Due to our misunderstanding of the submission requirements, we submitted graphs that are not connected, which makes our system only rank 6th over 10 teams. However, after we fixed this problem, our system is 0.6 ELAS higher than the team that ranked 1st in the official results.

Introduction

Traditional dependency parsing typically results in a "tree" structure where every word has exactly one head. However, real-world language is more complex. Enhanced Universal Dependencies (EUD) go beyond trees to represent phenomena like shared subjects in control structures, cycles, and implied "empty nodes." This transition turns parsing into a graph prediction problem, presenting two primary challenges: structural complexity (reentrancies) and data scarcity for low-resource languages like Tamil, which provided only 400 training sentences.

Methodology

Our system approaches EUD parsing as a semantic dependency parsing (SDP) task, utilizing a three-stage architecture:

  • The Encoder: We use a combination of pre-trained embeddings—XLMR, Flair, and FastText—fed into a three-layer BiLSTM to create rich, contextualized word representations.
  • Second-Order Inference: Most parsers are "first-order," scoring potential edges independently. Our system uses Mean-Field Variational Inference (MFVI) to perform second-order inference, allowing the model to consider "sibling" and "grandparent" relationships between edges for more globally coherent graphs.
  • Data Mixing (The Tamil Strategy): To address the Tamil data shortage, we mixed Tamil data with high-resource languages (like Czech). Crucially, we removed the labels from the high-resource data. This allowed the model to learn general linguistic structures from the large dataset while learning specific labels only from the Tamil data.

Experimental Results

The system was evaluated on the IWPT 2020 Shared Task dataset across 17 languages using the ELAS (Enhanced Labeled Attachment Score) metric.

The Connectivity Paradox

A misunderstanding of submission requirements led to our initial submission of non-connected graphs, resulting in an official 6th-place ranking. Post-evaluation revealed that while non-connected graphs can score slightly higher in raw metrics, applying the Maximum Spanning Tree (MST) algorithm to ensure connectivity allowed our system to outperform the official 1st-place team by 0.56 ELAS on average.

Comparison of official IWPT 2020 results against post-evaluation performance showing the system's potential as the top-performing model.

Low-Resource Performance

Our data-mixing strategy proved highly effective for Tamil. By combining Tamil data with Czech, we achieved a substantial boost in ELAS, demonstrating that structural transfer is possible even when label sets do not match.

Impact of data mixing strategies on Tamil ELAS performance using donor languages like English and Czech.

Discussion and Conclusion

Our results demonstrate that second-order inference provides a significant performance advantage over first-order methods by considering edge interactions. Furthermore, the "Tamil Trick" shows that you don't need matching labels to help a low-resource language; stripping labels from a high-resource "donor" language still allows the model to learn valuable lessons about arc behavior.

While effective, our data-mixing approach currently relies on manual selection of donor languages. Future work will explore automated methods for selecting donor languages based on linguistic similarity and investigate the computational trade-offs of second-order inference in large-scale applications.

The code for this project is available at: https://github.com/Alibaba-NLP/MultilangStructureKD

Summary updated

Abstract

This paper presents the system used in our submission to the \textit{IWPT 2020 Shared Task}. Our system is a graph-based parser with second-order inference. For the low-resource Tamil corpus, we specially mixed the training data of Tamil with other languages and significantly improved the performance of Tamil. Due to our misunderstanding of the submission requirements, we submitted graphs that are not connected, which makes our system only rank \textbf{6th} over 10 teams. However, after we fixed this problem, our system is 0.6 ELAS higher than the team that ranked \textbf{1st} in the official results.
BibTeX
@inproceedings{wang2020enhanced,
  author = {Xinyu Wang and Yong Jiang and Kewei Tu},
  title = {{Enhanced Universal Dependency Parsing with Second-Order Inference and Mixture of Training Data}},
  booktitle = {IWPT},
  year = {2020},
  url = {https://openreview.net/forum?id=kUJzSldRwF},
}
Figure