The Relative Clauses AMR Parsers Hate Most

Xiulin Yang, Nathan SchneiderCOLING 2024

TL;DR

This paper investigates how well current Abstract Meaning Representation (AMR) parsers handle English relative clauses (RCs), a complex linguistic construction. The authors find that while sequence-to-sequence models generally outperform older compositional models, all parsers struggle with RCs, particularly oblique and reduced forms, indicating a significant challenge for semantic parsing.

What Problem Does This Paper Address?

Abstract Meaning Representation (AMR) is a popular framework for semantic parsing, aiming to capture the meaning of sentences in a graph-like structure. While AMR parsers have achieved impressive overall scores, their performance on specific linguistic phenomena, especially those involving long-distance dependencies (LDDs), remains less understood. Relative clauses (RCs) are a prime example of LDDs, where a noun is modified by a clause that can be syntactically distant from it. These constructions can introduce complex semantic reentrancies (where a single node in the AMR graph represents multiple semantic roles or entities) that are crucial for accurate meaning representation. This paper aims to pinpoint how well existing AMR parsers can recover these RC-triggered reentrancies and identify which types of RCs pose the greatest challenge.

Methods

The researchers evaluated five AMR parsers, categorizing them into two groups: "structure-aware" and "structure-unaware." Structure-unaware models treat input as sequential strings, while structure-aware models are designed to incorporate structural information. The evaluation focused on measuring the "reentrancy recall" of RCs, which means assessing whether the parser correctly identifies the semantic connections introduced by the relative clause. This involves normalizing AMR graphs to make RC reentrancies more apparent and then checking if the head noun of the relative clause receives incoming edges from both the main clause predicate and the relative clause predicate. For parsing, they used a combination of existing AMR parsers and fine-tuned pre-trained language models.

Datasets

The study utilized two datasets:

  • English Web Treebank (EWT): A large, naturally occurring corpus of web text, annotated with Universal Dependencies (UD). The authors extracted sentences containing RCs from the training split.
  • Controlled RCs (CRC): A synthetically generated dataset designed to provide a balanced set of RC constructions and avoid confounds like length. This dataset includes specific types of RCs like subject, object, reduced object, and passive RCs.

The paper also mentions the AMR 3.0 dataset for reporting standard AMR parsing metrics.

Key Insights and Ideas

  • RCs are a major pain point for AMR parsers: Despite advancements in AMR parsing, relative clauses remain a significant challenge, with recall scores often falling below 64% on the EWT corpus.
  • Sequence-to-sequence models lead the pack: Modern seq2seq models, regardless of whether they incorporate structural biases, generally outperform older compositional AMR parsers.
  • Structure-awareness isn't a silver bullet (yet): The study found little performance difference between structure-aware and structure-unaware seq2seq models for RC parsing, suggesting that current methods for incorporating structure might not be fully effective for this specific task.
  • Not all RCs are created equal (for parsers): Passive subject RCs are the easiest for parsers to handle, while oblique and reduced RCs are the most challenging. This suggests that the syntactic complexity and explicitness of the relative clause structure play a crucial role in parsing difficulty.
  • Reduced RCs are surprisingly hard: Despite often having shorter dependency distances, reduced RCs (where the relative pronoun is omitted) are more difficult for parsers than their full counterparts. This might be due to the lack of explicit syntactic cues.

Key Results

  • Overall Recall: On the EWT corpus, recall for RC reentrancies was below 64% for all parsers.
  • Model Comparison: Seq2seq models generally outperformed the compositional AM-Parser. AMRBART and amrlib-BART showed strong performance among the seq2seq models.
  • RC Type Performance (EWT): Passive RCs had the highest recall (around 78%), while oblique RCs had the lowest (around 55.2%). Reduced object RCs also performed poorly (around 56.6%).
  • RC Type Performance (CRC): Scores on the synthetic CRC dataset were generally higher, with amrlib-BART showing a slight edge.
  • Dependency Distance: While shorter dependency distances in full RCs generally correlate with easier parsing, reduced RCs with the shortest distances were still challenging, suggesting other factors are at play.

Limitations and Future Work

  • Alignment Challenges: The study acknowledges that the accuracy of their evaluation metric is dependent on the quality of token-to-AMR alignments, and misalignments can affect scores.
  • Limited Scope of Structure-Awareness: The lack of significant performance gains from structure-aware models suggests that current methods for incorporating structure might need further refinement.
  • Future Directions: The authors suggest expanding the analysis to other reentrancy types and exploring whether incorporating Enhanced Universal Dependencies (EUD) information can improve the ability of structure-unaware parsers to learn complex structural information. They also plan to release their gold EUD annotations for reduced RCs.

Summary updated

BibTeX
@inproceedings{yang2024relative,
  author = {Xiulin Yang and Nathan Schneider},
  title = {{The Relative Clauses AMR Parsers Hate Most}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2024},
}
Figure