Multitask Learning for Cross-Lingual Transfer of Broad-coverage Semantic Dependencies

Maryam Aminian, Mohammad Sadegh Rasooli, Mona T. DiabEMNLP 2020 · OpenReview

Abstract

We describe a method for developing broad-coverage semantic dependency parsers for languages for which no semantically annotated resource is available. We leverage a multitask learning framework coupled with annotation projection. We use syntactic parsing as the auxiliary task in our multitask setup. Our annotation projection experiments from English to Czech show that our multitask setup yields 3.1% (4.2%) improvement in labeled F1-score on in-domain (out-of-domain) test set compared to a single-task baseline.

Introduction

Semantic Dependency Parsing (SDP) is the task of identifying "who did what to whom" in a sentence, representing these relationships as a rich graph. Unlike traditional syntactic trees, these graphs can capture complex phenomena like negation and multiple roles for a single word. However, creating gold-standard annotations is incredibly expensive, and high-quality SDP data currently exists for only a few languages. This research addresses this data scarcity by asking: Can we transfer semantic knowledge from English to another language without any native semantic labels, and can we use easier-to-find syntactic data to help?

Methods

The proposed approach combines two key strategies:

  1. Annotation Projection: Using a large parallel corpus (Europarl), English sentences are parsed with a high-quality parser. These labels are then "projected" onto the target language (Czech) using automated word-alignment tools. This creates a "noisy" but usable training set.
  2. Multitask Learning (MTL): Recognizing that syntax and semantics are deeply intertwined, the authors built a neural model that performs both tasks simultaneously. The architecture uses shared bidirectional LSTM (BiLSTM) layers to capture features relevant to both syntactic and semantic structures, followed by task-specific Biaffine scorers.
The multitask learning architecture utilizes shared BiLSTM layers to capture features relevant to both syntactic and semantic parsing.

Key Insights and Results

The multitask model achieved a 3.1% absolute improvement in labeled F1-score for in-domain Czech and a 4.2% improvement for out-of-domain data compared to a single-task baseline.

  • Syntax as a Scaffold: Even with noisy projected semantic labels, the model benefits immensely from learning "clean" syntax. Syntax provides a structural backbone that helps the model resolve semantic ambiguities and generalize better to unfamiliar domains.
  • Long-Distance Success: The MTL approach is particularly effective at capturing long-distance dependencies. Analysis shows that the MTL model significantly outperforms the single-task baseline for dependencies spanning 10 or more tokens.
The MTL model shows significant improvements in labeled precision for dependencies spanning 10 or more tokens compared to the single-task baseline.
  • Impact of Contextual Embeddings: Interestingly, adding BERT or ELMo provided only marginal gains. This suggests that in a cross-lingual projection setting, the "noise" in the labels might be a more significant bottleneck than the quality of the word representations.

Conclusion

This work demonstrates that multitask learning, coupled with annotation projection, is a viable path for developing semantic parsers in low-resource settings. While still trailing behind fully supervised models, this approach provides a respectable "cold start" for languages with zero native semantic annotations. Future research will explore more sophisticated noise-handling techniques and broader language coverage.

Summary updated

Abstract

We describe a method for developing broad-coverage semantic dependency parsers for languages for which no semantically annotated resource is available. We leverage a multitask learning framework coupled with annotation projection. We use syntactic parsing as the auxiliary task in our multitask setup. Our annotation projection experiments from English to Czech show that our multitask setup yields 3.1% (4.2%) improvement in labeled F1-score on in-domain (out-of-domain) test set compared to a single-task baseline.
BibTeX
@inproceedings{aminian2020multitask,
  author = {Maryam Aminian and Mohammad Sadegh Rasooli and Mona T. Diab},
  title = {{Multitask Learning for Cross-Lingual Transfer of Broad-coverage Semantic Dependencies}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=agXqkC95yz},
}
Figure