Biaffine Dependency and Semantic Graph Parsing for EnhancedUniversal Dependencies

Giuseppe Attardi, Daniele Sartiano, Maria SimiACL 2021

TL;DR

This paper introduces a system for enhanced Universal Dependencies (UD) parsing that combines a biaffine dependency parser with a semantic graph parser. Their approach achieves competitive performance on the IWPT 2021 Shared Task, particularly for languages with ample data, and explores a promising multilingual model for low-resource scenarios.

What Problem Does This Paper Address?

The paper tackles the challenge of enhanced Universal Dependencies parsing, which goes beyond standard dependency parsing by aiming to represent richer semantic relationships in text. The goal is to convert raw text into a structured representation that captures not just syntactic dependencies but also semantic roles and connections. This is particularly difficult for low-resource languages where training data is scarce, and for achieving high accuracy across a diverse set of languages. The IWPT 2021 Shared Task specifically focused on this multi-language, enhanced parsing challenge.

Methods

The core of the system relies on two main components: a biaffine dependency parser and a semantic graph parser. Both parsers share a similar neural architecture, leveraging transformers for contextualized word representations.

  • Dependency Parser: This component uses a biaffine attention mechanism to predict dependency arcs and their labels. A novel aspect is the incorporation of "hints" derived from the transformer's attention layers, specifically from layer 6, which are added to the biaffine layer's score. This aims to guide the parser by leveraging structural information implicitly learned by the transformer. The Chu-Liu-Edmonds algorithm is used to ensure the resulting parse trees are well-formed (i.e., acyclic).
  • Semantic Graph Parser: This parser shares the same architecture but uses a sigmoid cross-entropy loss function instead of softmax. This allows for multiple semantic relationships (arcs) to be predicted for a given pair of words, reflecting the potentially richer nature of semantic graphs compared to syntactic trees. Arcs with a logit value greater than zero are retained.
  • Merging Outputs: The final enhanced dependency output is obtained by merging the predictions from both the syntactic and semantic parsers. This is done by taking the union of the predicted arcs, with a check to ensure no extra arcs to the root are introduced.

A significant experimental effort was also dedicated to a pretrained multilingual model. This involved concatenating training corpora from all languages, de-lexicalizing the enhanced dependency labels (replacing specific words with placeholders), and training a single model using XLM-Rooberta. This multilingual model was then fine-tuned on each language.

Datasets

The experiments were conducted on treebanks for seventeen languages participating in the IWPT 2021 Shared Task. The paper mentions using official train and gold development sets for hyperparameter tuning and evaluation. Specific mention is made of the Italian UD treebank (Italian_ISST) for training a custom multi-word tokenization model.

Key Insights and Ideas

  • Transformer Attention as Structural Hints: Using attention weights from specific transformer layers as "hints" for the biaffine parser proved effective, leading to a small but noticeable improvement in performance on English. This suggests that transformers implicitly learn valuable structural information that can be explicitly leveraged.
  • Sigmoid for Semantic Graphs: The use of a sigmoid cross-entropy loss for the semantic graph parser is a key adaptation, allowing for the prediction of multiple semantic relations, which is more aligned with the nature of semantic graphs than a strict tree structure.
  • Merging Parsers for Robustness: Combining the outputs of a syntactic and a semantic parser offers a robust way to generate enhanced dependency representations, leveraging the strengths of both approaches.
  • De-lexicalization for Multilingual Models: De-lexicalizing dependency labels is a crucial preprocessing step for building effective multilingual models, as it reduces the label space and allows the model to generalize better across languages, especially for low-resource scenarios.

Key Results

  • The combined dependency and semantic graph parser achieved competitive LAS (Labeled Attachment Score) performance on the IWPT 2021 Shared Task, often ranking high among systems reporting results on this metric.
  • On the English development corpus, the extension of the biaffine parser with transformer attention hints improved UAS (Unlabeled Attachment Score) to 92.21 and LAS to 90.31, compared to 91.32 UAS and 89.33 LAS without these features.
  • The multilingual model showed promising improvements for languages with smaller corpora, notably Lithuanian, which saw a 6.75 point increase in EULAS (Enhanced Universal Dependency Labeled Attachment Score).

Limitations and Future Work

  • Low-Resource Language Performance: While the multilingual model showed promise, the paper acknowledges that ELAS scores did not improve as much as expected, suggesting that the ri-lexicalization algorithms might need further tuning for each language.
  • Merging Strategy: The paper mentions that the merging strategy does not guarantee a fully connected graph, and the final output is obtained by merging with the syntactic parser's tree.
  • Future Work: The authors suggest that performing joint training of the syntactic and semantic parsers is a promising direction, given their similar architectures. Improving the back-conversion algorithm for introducing lexical information in the multilingual model is also highlighted as an area for future research.

Summary updated

BibTeX
@inproceedings{attardi2021biaffine,
  author = {Giuseppe Attardi and Daniele Sartiano and Maria Simi},
  title = {{Biaffine Dependency and Semantic Graph Parsing for EnhancedUniversal Dependencies}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure