Annotations Matter: Leveraging Multi-task Learning to Parse UD and SUD

Zeeshan Ali Sayyed, Daniel DakotaACL 2021

TL;DR

This paper investigates how different syntactic annotation schemes (Universal Dependencies vs. Surface-Syntactic Universal Dependencies) impact multi-task learning (MTL) for dependency parsing. They found that while MTL can leverage multiple treebanks, competing annotation decisions can lead to negative transfer, resulting in worse performance than single-task baselines. This highlights that the nature of annotations, not just the amount of data, is crucial for successful MTL.

What Problem Does This Paper Address?

Multi-task learning (MTL) is a popular technique in NLP to improve model performance by training on multiple related tasks simultaneously. The assumption is that sharing information across tasks leads to better generalization. However, it's not always clear what information should be shared or when MTL will be beneficial. This paper specifically tackles the question of how different annotation schemes within the same linguistic domain (dependency parsing) affect MTL performance. They focus on Universal Dependencies (UD) and Surface-Syntactic Universal Dependencies (SUD), which, despite being similar, have distinct annotation preferences. The core problem is understanding if and why these annotation differences can hinder MTL, leading to negative transfer, rather than the expected positive transfer.

Methods

The researchers adapted the graph-based Deep Biaffine Parser (Dozat and Manning, 2017) into a multi-task learning architecture. Each treebank (UD and SUD) was treated as a separate task. They experimented with two main architectural choices:

  • Shared vs. Unshared MLP Layers: MLPs (Multi-Layer Perceptrons) were used to process BiLSTM encodings. Sharing these layers aimed to see if minimizing irrelevant information across tasks would improve performance. Unshared layers allowed each task to have its own processing.
  • Joint vs. Alternating Loss: They explored two loss functions:
  • Joint Loss: The model optimizes the sum of losses from both tasks simultaneously.
  • Alternating Loss: The model randomly picks one task at a time to optimize.

They also experimented with various input embeddings, including randomly initialized word embeddings, FastText, character embeddings, POS tags, and BERT, and their combinations.

Datasets

The primary datasets used were:

  • Universal Dependencies (UD): A widely used, standardized treebank collection for dependency parsing.
  • Surface-Syntactic Universal Dependencies (SUD): A newer annotation scheme that aims to be more syntactically motivated, often resulting in different tree structures compared to UD, particularly in terms of non-projectivity.

They selected 12 languages from 10 language families for UD and SUD. Additionally, they used a subset of treebanks from the SPMRL Shared Task (Arabic, German, Hungarian) to investigate the impact of differently annotated treebanks when paired with UD/SUD in an MTL setup.

Key Insights and Ideas

  • Annotations Matter More Than Embeddings: The most striking finding is that the nature of the syntactic annotations themselves plays a more significant role in MTL performance than the choice of input embeddings. Competing annotation decisions can introduce "mixed signals" that hamper the MTL model.
  • Negative Transfer is Real: Contrary to the expectation that MTL always improves performance, this study demonstrates that competing annotation schemes can lead to negative transfer, resulting in performance drops compared to single-task baselines.
  • UD vs. SUD: A Case Study in Annotation Conflict: The UD and SUD schemes, while related, exhibit differences (e.g., in head selection, leading to more non-projective trees in SUD) that create inherent conflicts when trained together in an MTL setup.
  • POS Embeddings as a Mitigator: While character embeddings, which are beneficial for single-task parsers, proved detrimental in the MTL setting, POS tag embeddings seemed to help mitigate performance loss. This suggests that POS tags might provide more universally beneficial signals for both annotation schemes.
  • SPMRL Experiments Show Promise: When pairing UD/SUD with differently annotated treebanks from the SPMRL dataset, they observed performance increases over baselines, suggesting that compatibility and complementary information between treebanks are key for successful MTL.

Key Results

  • MTL Performance Drop: In general, MTL models trained on UD and SUD performed worse than their single-task baselines across most languages and embedding configurations. The mean drop in LAS scores for MTL settings compared to baselines was around 0.70-0.76.
  • Joint Loss vs. Alternating Loss: Joint loss converged faster in terms of time, though it required more epochs. The performance differences between joint and alternating loss were minimal.
  • Shared vs. Unshared MLPs: Sharing MLP layers showed a slight advantage, indicating that removing task-specific irrelevant information was beneficial.
  • Embedding Impact: Randomly initialized word embeddings led to noticeably worse performance. Combinations of embeddings generally performed better than single embeddings, with the concatenation of all embeddings yielding the best results across MTL settings. However, even with the best embeddings, the overall MTL performance was still below baselines when using UD and SUD.
  • SPMRL Results: Experiments with SPMRL treebanks showed improved performance over baselines, particularly with character-based embeddings, suggesting that compatible annotation schemes are crucial for MTL gains.

Limitations and Future Work

  • Limitations:
    • The study primarily focuses on dependency parsing; the findings might not generalize to other NLP tasks.
  • The performance degradation observed in UD-SUD MTL settings suggests that the specific annotation differences are the primary cause, rather than just the amount of data.
  • The paper notes that punctuation attachment errors increased significantly in MTL, highlighting how even seemingly minor annotation differences can create systematic issues.
  • Future Work:
    • Incorporate more treebanks with diverse annotation schemes to explore how different annotation directions influence MTL.
  • Investigate leveraging constituency parsing and dependency parsing together in MTL setups.
  • Further explore the role of embedding choices and their interaction with annotation schemes.

Summary updated

BibTeX
@inproceedings{sayyed2021annotations,
  author = {Zeeshan Ali Sayyed and Daniel Dakota},
  title = {{Annotations Matter: Leveraging Multi-task Learning to Parse UD and SUD}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure