CAMR at SemEval-2016 Task 8: An Extended Transition-based AMR Parser

Chuan Wang, Sameer Pradhan, Xiaoman Pan, Heng Ji, Nianwen XueSemEval @ NAACL 2016 · OpenReview

TL;DR

This paper presents CAMR, an extended transition-based parser for Abstract Meaning Representation (AMR). By incorporating rich named entities, a verbalization list, and semantic role labels as features, they significantly improve AMR parsing accuracy, achieving a Smatch F-score of 62% on the SemEval-2016 Task 8 blind test set.

What Problem Does This Paper Address?

Abstract Meaning Representation (AMR) is a powerful way to represent the meaning of sentences as directed, edge-labeled graphs. However, accurately parsing sentences into AMR graphs is a challenging task, and existing AMR parsers are generally less accurate than syntactic parsers. This lower accuracy hinders their usefulness in downstream Natural Language Processing (NLP) tasks. The authors aim to boost AMR parsing performance by exploring additional sources of information that can enrich the parsing model.

Methods

The core of the system is CAMR, a transition-based AMR parser that operates in two stages. First, it parses the input sentence into a dependency tree using an off-the-shelf dependency parser. Second, it transforms this dependency tree into an AMR graph through a series of carefully designed actions.

The paper's main contribution lies in the extensions made to this existing CAMR parser for the SemEval-2016 task. They introduce three key sets of new features:

  1. Rich Named Entities: AMR concepts often have more fine-grained named entity types than standard NER systems. They leverage the 18 named entity types from the OntoNotes v5.0 Corpus to improve concept identification.
  2. Verbalization List: Many AMR concepts are "normalized" English words, where a verb form is used for a noun or adjective (e.g., "attractive" maps to "attract-01"). They use a verbalization list provided by ISI to better align words to these normalized AMR concepts.
  3. Semantic Role Labels: They incorporate semantic role labeling features, specifically checking if a predicted concept label matches a frameset predicted by ASSERT (a semantic role labeler) and if a dependent is predicted as an argument of the current head.

Additionally, they employ a wikification system (based on Pan et al., 2015) to resolve entity mentions to their corresponding Wikipedia entries, which is a novel aspect of the SemEval-2016 release and aids in scoring.

Datasets

The experiments are conducted using the SemEval-2016 Task 8 dataset, which is based on the LDC2015E86 release. This release is notable for including wikification information, which was absent in previous AMR releases (LDC2013E117 and LDC2014T12). They also use the LDC2014T12 release for additional experiments to isolate the impact of certain features.

Key Insights and Ideas

  • Feature Engineering is Crucial: The paper strongly emphasizes that augmenting existing parsers with rich, external linguistic information (like named entities, verbalizations, and semantic roles) is a highly effective strategy for improving AMR parsing performance.
  • Fine-Grained NER Matters: The use of richer named entity types from OntoNotes, compared to standard NER, provides a tangible benefit for concept identification in AMR.
  • Bridging Lexical Gaps: The verbalization list is key to handling the common practice in AMR of using normalized verb forms for concepts derived from nouns or adjectives, a common source of error.
  • Wikification as a Post-Processing Step: While not directly part of the parsing process, the authors found that a well-designed wikifier can be a helpful post-processing step, especially given its integration into the SemEval evaluation metric.
  • Brown Clusters Can Hurt: Surprisingly, the Brown cluster features, often beneficial in other NLP tasks, actually degraded performance in this specific AMR parsing setup, suggesting that feature interactions can be complex.

Key Results

The paper reports significant improvements over their baseline CAMR parser. On the official SemEval development set (LDC2015E86), adding the verbalization list, semantic role labels, and wikification individually led to improvements. The combination of these features, along with rich named entities, resulted in a Smatch F-score of 66.56% on the development set.

On the SemEval test set, their parser achieved an F-score of 66.54%. For the SemEval blind test set, they report a Smatch F-score of 62.01%. This is a substantial gain, demonstrating the effectiveness of their extended feature set. They also note that their parser remained relatively stable between the test and blind test sets, though the blind test set proved harder.

Limitations and Future Work

  • Blind Test Set Performance Drop: The evaluation on the blind test set showed a drop of around 4 points compared to the test set. The authors acknowledge this and plan to conduct further error analysis to understand the reasons for this difference.
  • Brown Clusters Detrimental: The negative impact of Brown cluster features suggests that further investigation into feature selection and interaction is warranted.
  • Wikification Integration: While beneficial, the wikification is presented as a post-processing step. Future work could explore tighter integration of wikification information directly into the parsing model.
  • Generalizability: While the results are strong, further evaluation on different AMR datasets and domains would be valuable to assess the generalizability of their approach.

Summary updated

BibTeX
@inproceedings{wang2016camr,
  author = {Chuan Wang and Sameer Pradhan and Xiaoman Pan and Heng Ji and Nianwen Xue},
  title = {{CAMR at SemEval-2016 Task 8: An Extended Transition-based AMR Parser}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  url = {https://openreview.net/forum?id=tvOisfqaOEE},
}
Figure