DRS Parsing as Sequence Labeling

Minxing Shen, Kilian EvangNAACL 2022

TL;DR

This paper introduces a novel approach to semantic parsing by framing Discourse Representation Structure (DRS) generation as a sequence labeling task. This method achieves state-of-the-art accuracy across multiple languages while offering improved interpretability and compositionality compared to traditional sequence-to-sequence models.

What Problem Does This Paper Address?

Semantic parsing aims to translate natural language sentences into formal, symbolic representations of their meaning. While end-to-end systems dominate many Natural Language Understanding (NLU) applications, symbolic representations offer advantages for reasoning, explainability, and debugging. Discourse Representation Structures (DRSs) are a powerful formalism for capturing meaning, especially aspects like negation and quantification. However, existing DRS parsers often rely on complex architectures and lack the desirable property of compositionality, where the meaning of a whole is a function of the meanings of its parts. This makes them harder to understand and integrate into human-in-the-loop annotation workflows. The authors address this by developing a trainable DRS parser that is both accurate and compositional.

Methods

The core innovation is framing DRS parsing as a sequence labeling task. Instead of generating a complex DRS structure directly, the system assigns a finite set of "meaning fragments" to each token in the input sentence. These fragments are then deterministically combined to form the final DRS.

Key methodological aspects include:

  • Anchored DRS Encoding: DRS clauses are aligned to input tokens, allowing for a token-level representation.
  • Sequence Labeling: A standard transformer-based model (BERT) is used to predict labels for each token. These labels can include word senses, fragments, and integration labels.
  • Relative Referent Encoding: To improve generalization, referent indices in the labels are made relative, indicating how long ago a referent of the same type last appeared.
  • Fragment Factorization: Fragments are further broken down into "fragments proper" and "integration labels" to reduce the number of unique fragments.
  • Post-processing: A set of rules is applied to ensure the generated DRSs are well-formed and to handle aspects like symbol prediction (names, dates) and pronoun resolution.

Datasets

The authors train and evaluate their models on the Parallel Meaning Bank (PMB) version 3.0.0. This dataset contains sentences annotated with anchored DRSs in four languages: English, German, Italian, and Dutch. The dataset includes annotations with different quality levels: gold (fully corrected), silver (partially corrected), and bronze (rule-based pre-annotation). For Italian and Dutch, due to limited gold data, they primarily use silver and bronze data for training.

Key Insights and Ideas

  • Compositionality as a Practical Goal: The paper demonstrates that achieving a degree of compositionality in a neural semantic parser is feasible and beneficial, particularly for interpretability and human-in-the-loop scenarios.
  • Sequence Labeling for Complex Structures: Framing complex structured prediction tasks like DRS parsing as sequence labeling simplifies the model architecture and leverages powerful pre-trained language models effectively.
  • Relative Referents Improve Generalization: Making referent indices relative, rather than absolute, significantly helps the model generalize to unseen sentences by reducing reliance on sentence-specific referent counts.
  • Trade-offs in Fragment Factorization: While factoring fragments into "proper" and "integration" labels reduces the number of unique fragments, the integration labels themselves can be highly unbalanced, impacting overall performance.
  • BERT's Power for Semantic Tasks: The use of pre-trained BERT embeddings significantly boosts performance, and fine-tuning BERT parameters is found to be more effective than keeping them frozen.

Key Results

  • Competitive Accuracy: The proposed DRS parser achieves accuracy competitive with state-of-the-art systems on the PMB dataset, particularly for English. For example, their best model on English achieves a Counter f-score of 88.4, very close to the state-of-the-art.
  • Cross-lingual Performance: The model demonstrates good performance across German, Italian, and Dutch, though performance generally aligns with the amount of available training data.
  • Improved Interpretability: The token-to-fragment mapping provides a more transparent output compared to black-box sequence-to-sequence models.
  • Error Analysis Highlights Challenges: The analysis reveals specific phenomena that are challenging, including NP coordination with three or more conjuncts, temporal expressions, cardinalities, named entities, universal quantification, and rhetorical relations. Word senses and semantic roles are also identified as common error sources.

Limitations and Future Work

  • Post-processing Heuristics: The system relies on some post-processing heuristics to ensure valid DRSs, which are not strictly part of the compositional framework.
  • Limited Handling of Complex Meaning Layers: The current model does not explicitly handle partially compositional or non-compositional layers of meaning like scope or anaphora.
  • Discourse Context: The paper suggests that modeling discourse context more effectively is a key area for future work, particularly for resolving ambiguities like generic vs. deictic "you."
  • Sentence vs. Speaker Meaning: Explicitly distinguishing between sentence meaning and speaker meaning is identified as a potential avenue for improvement.
  • Realignment for Conjunctions: The authors suggest that a realignment step, similar to what's used for pronouns, could improve handling of conjunctions.
  • Specialized Neural Transcoders: Developing specialized neural transcoders could improve predictions for open-class strings like names and dates.

Summary updated

BibTeX
@inproceedings{shen2022drs,
  author = {Minxing Shen and Kilian Evang},
  title = {{DRS Parsing as Sequence Labeling}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}

Loading related content...

Figure