Joint Universal Syntactic and Semantic Parsing

Elias Stengel-Eskin, Kenton W. Murray, Sheng Zhang, Aaron Steven White, Benjamin Van Durme

TL;DR

This paper tackles the long-standing challenge of jointly parsing syntactic and semantic information, where previous attempts often saw performance trade-offs. By leveraging the Universal Decompositional Semantics (UDS) dataset, which pairs syntactic (Universal Dependencies) and semantic annotations, the authors develop novel Transformer-based architectures that achieve state-of-the-art results in both domains simultaneously, demonstrating a mutually beneficial relationship.

What Problem Does This Paper Address?

For decades, researchers have recognized the deep connection between syntax (the structure of sentences) and semantics (their meaning). Intuitively, a model that understands both should perform better than one that focuses on only one aspect. However, in practice, building models that excel at both syntactic and semantic parsing has been difficult. Often, improving performance in one area came at the cost of performance in the other. This paper aims to break this trade-off by developing models that can effectively exploit the rich, intertwined information present in syntactic and semantic annotations, specifically using the Universal Decompositional Semantics (UDS) dataset.

Methods

The core of this work involves developing and evaluating novel neural network architectures for joint syntactic and semantic parsing. They build upon a previous transductive sequence-to-graph parsing model but introduce significant improvements:

  • Transformer-based Architectures: They replace the original LSTM encoders and decoders with Transformer-based architectures, which are known for their ability to capture long-range dependencies.
  • Joint Syntactic and Semantic Objectives: The key innovation is the simultaneous training of models for both Universal Dependencies (UD) syntactic parsing and Universal Decompositional Semantics (UDS) parsing. This is achieved through various multitask learning strategies.
  • Encoder-Side Syntactic Parsing: A novel approach involves adding a biaffine syntactic parser that operates on the encoder's output. This allows the model to learn syntactic structure concurrently with semantic parsing, even if the syntactic signal isn't directly fed into the decoder.
  • Intermediate Syntactic Representation: Another method involves re-encoding the predicted syntactic parse and feeding it into the decoder, allowing the semantic component direct access to syntactic information.
  • Multilingual Extension: They leverage the multilingual capabilities of models like XLM-R to explore how joint parsing benefits extend to non-English languages.

Datasets

  • English Web Treebank (EWT): This dataset, containing both Universal Dependencies (UD) syntactic annotations and Universal Decompositional Semantics (UDS) annotations, is central to the paper. The authors use a semantically valid subset of EWT.
  • UDS Dataset: The Universal Decompositional Semantics (UDS) dataset is specifically highlighted for its rich, parallel syntactic and semantic annotations, including scalar-valued attributes for semantic inferences.
  • Multilingual UD Corpora: For multilingual experiments, they use data from 8 languages across 5 families, sourced from the 2018 CoNLL Shared Task.

Key Insights and Ideas

  • Syntax and Semantics are Truly Complementary: The paper provides strong evidence that syntactic and semantic signals are not only beneficial to each other but also complementary to the information captured by pre-trained contextualized encoders (like BERT). This suggests that even powerful language models don't fully capture all the nuances of syntax-semantics interaction.
  • Transformer Architectures Excel: Transformer-based models consistently outperform LSTM-based models across various metrics, highlighting their effectiveness in capturing the complex relationships required for joint parsing.
  • The "Intermediate" Model's Promise: While the encoder-side syntactic parsing shows strong results, the "intermediate" model (where syntactic information is re-encoded and fed to the decoder) shows potential for even greater gains, especially in multilingual settings, suggesting future avenues for research.
  • Multilingual Benefits: The positive effects of joint parsing are not limited to English; similar trends are observed across multiple languages, indicating that the learned representations capture universal linguistic principles.
  • Syntactic Signal Aids Disambiguation: The paper explores the hypothesis that semantic information can help resolve syntactic ambiguities (like prepositional phrase attachment), finding some evidence for this, though the effect is subtle.

Key Results

  • State-of-the-Art Performance: The proposed joint Transformer-based models achieve state-of-the-art results on both UD syntactic parsing and UDS semantic parsing benchmarks.
  • Significant Gains: The joint models outperform previous best results by a large margin, particularly in UDS performance. For example, they achieve UAS/LAS scores of 93.42 and 91.22 on a cleaned EWT subset, surpassing previous benchmarks.
  • Syntactic Improvement from Semantics: The paper demonstrates a "top-down" benefit, where semantic information improves syntactic parsing performance, especially in multilingual settings.
  • Semantic Improvement from Syntax: Conversely, a "bottom-up" benefit is shown, where syntactic information enhances semantic parsing performance.
  • Attribute Prediction: Transformer models show a significant advantage in predicting UDS attributes, likely due to their superior ability to capture context from across the entire input sentence.

Limitations and Future Work

  • Data Sparsity and Overfitting: In experiments tuning BERT encoder layers, the authors observed that unfreezing too many layers led to decreased performance, suggesting potential overfitting due to data sparsity.
  • "Intermediate" Model Performance: While promising, the "intermediate" model's performance was sometimes lower than the encoder-side model, especially in monolingual settings, indicating room for further architectural refinement.
  • Prepositional Phrase Attachment Ambiguity: While the paper investigates this, the observed benefits of joint parsing for disambiguating PP attachment were small, suggesting this is not the primary driver of performance gains.
  • Low-Resource Settings: The authors suggest improving the robustness of Transformer models in low-resource settings as a future direction.
  • Attribute Modules: Further improvements to the attribute prediction modules are also identified as a promising area for future work.
  • Expanding UDS Data: Building robust multilingual parsing models by expanding the UDS data paradigm beyond English is a key recommendation for future research.

Summary updated

Abstract

While numerous attempts have been made to jointly parse syntax and semantics, high performance in one domain typically comes at the price of performance in the other. This trade-off contradicts the large body of research focusing on the rich interactions at the syntax-semantics interface. We explore multiple model architectures which allow us to exploit the rich syntactic and semantic annotations contained in the Universal Decompositional Semantics (UDS) dataset, jointly parsing Universal Dependencies and UDS to obtain state-of-the-art results in both formalisms. We analyze the behaviour of a joint model of syntax and semantics, finding patterns supported by linguistic theory at the syntax-semantics interface. We then investigate to what degree joint modeling generalizes to a multilingual setting, where we find similar trends across 8 languages.
BibTeX
@article{stengeleskin2021joint,
  author = {Elias Stengel-Eskin and Kenton W. Murray and Sheng Zhang and Aaron Steven White and Benjamin Van Durme},
  title = {{Joint Universal Syntactic and Semantic Parsing}},
  journal = {Trans. Assoc. Comput. Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=Z_x1r7BMRch},
}
Figure