Parsing Meaning Representations: Is Easier Always Better?

Zi Lin, Nianwen XueACL 2019

TL;DR

This paper investigates why parsing Abstract Meaning Representation (AMR) is significantly harder than parsing Minimal Recursion Semantics (MRS). By comparing parsing performance on the same parser and evaluation metric, the authors pinpoint key differences in how AMR and MRS represent linguistic information, such as concept abstraction and semantic role definition, as the primary drivers of this discrepancy.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) aims to represent the meaning of sentences in a formal, machine-readable way. This is crucial for downstream tasks like question answering and machine translation. However, there's no single, universally accepted "meaning representation" (MR). Instead, various MRs exist, each with different design choices regarding what aspects of meaning to capture and how to encode them. This paper focuses on two prominent MRs: Abstract Meaning Representation (AMR) and Minimal Recursion Semantics (MRS). While both are graph-based, empirical results show a substantial gap in parsing accuracy, with MRS being significantly easier to parse automatically. The core problem this paper addresses is understanding why this gap exists and identifying the specific design choices in AMR that make it more challenging to parse compared to MRS. This understanding is vital for designing future, more effective meaning representations.

Methods

The authors employ a comparative approach to dissect the parsing difficulty. They use a single, transition-based parser, CAMR, originally developed for AMR, and adapt it to parse MRS. This ensures that differences in parsing performance are attributable to the MRs themselves, not the parsing algorithms. They also use the SMATCH metric for evaluation, ensuring consistency. The core of their methodology involves a detailed analysis of the parsing process, breaking it down into concept detection and relation detection. They analyze how different linguistic phenomena (e.g., phrasal verbs, named entities, prepositions) are represented in each MR and how these representations impact parsing accuracy.

Datasets

The experiments were conducted using the following datasets:

  • MRS parsing: SDP2015 dataset.
  • AMR parsing: LDC2016E25 dataset.

The paper notes that the number of sentences/graphs in both datasets is similar, which is important for a fair comparison. The MRS data comes with manual alignments between words and MR elements, while the AMR data required the use of an automatic aligner (JAMR) to establish these alignments.

Key Insights and Ideas

  • Abstraction is Key: AMR's higher level of abstraction from surface forms is a major contributor to parsing difficulty. AMR concepts often bear less resemblance to the original word tokens compared to MRS.
  • Fine-Grained Entity Classification: AMR's more detailed classification of named entities, while potentially richer, introduces more complexity and potential for errors during parsing.
  • Semantic Relation Differences: The way semantic relations are defined in AMR and MRS leads to different parsing challenges. AMR's roles are tied to specific predicate senses in PropBank, while MRS roles are more general and reflect obliqueness.
  • Prepositions and Coreference: AMR's handling of prepositions (often dropping them and making their objects arguments of the verb) and its lack of explicit coreference resolution (unlike some MRS variants) simplify parsing for MRS.
  • "Necessary Evils" vs. Design Choices: The paper aims to distinguish between challenges that are inherent to capturing rich meaning (potentially "necessary evils") and those that arise from specific design choices that could be simplified without sacrificing utility.

Key Results

The study found a significant difference in parsing performance, with the SMATCH score for MRS parsing being approximately 20% higher than for AMR. Specifically:

  • Overall SMATCH Score: The paper reports a gap of 17.3% in F1 score between MRS and AMR parsing using CAMR.
  • Concept Detection: AMR's overall concept detection accuracy is much lower than MRS. However, the difference is small for concepts requiring word sense disambiguation, suggesting this is not the primary driver of the gap.
  • Entity Detection: AMR's date entity detection accuracy is significantly lower than MRS. While AMR has higher accuracy for named entity detection, the additional step of classification makes overall named entity parsing harder.
  • Relation Detection (SRL): Accuracy for Semantic Role Labeling (SRL) is considerably lower for AMR than MRS. This is attributed to differences in how prepositions and coreference are handled, and the definition of semantic roles.

Limitations and Future Work

  • Data Alignment: The paper acknowledges that the lack of manual alignment for the AMR dataset and potential inconsistencies in annotation could contribute to the performance gap.
  • Parser Adaptation: While CAMR was adapted for MRS, it was originally designed for AMR. Future work could explore parsers specifically designed for MRS or cross-framework parsers that are more balanced.
  • Future MR Design: The authors hope their findings will inform the design of future meaning representations, suggesting a need to carefully consider the trade-off between expressiveness and ease of automatic parsing. They aim to help researchers make more informed decisions about which aspects of meaning are crucial and how best to represent them for computational processing.

Summary updated

BibTeX
@inproceedings{lin2019parsing,
  author = {Zi Lin and Nianwen Xue},
  title = {{Parsing Meaning Representations: Is Easier Always Better?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure