AMR Beyond the Sentence: the Multi-sentence AMR corpus

Tim O’Gorman, Michael Regan, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Martha PalmerCOLING 2018

TL;DR

This paper introduces the Multi-Sentence AMR (MS-AMR) corpus, a novel dataset that extends Abstract Meaning Representation (AMR) to capture document-level semantics by annotating coreference, implicit roles, and bridging relations across sentences. This work addresses the limitation of existing AMR, which is sentence-bound, by enabling richer representations of discourse meaning crucial for tasks like information extraction and question answering.

What Problem Does This Paper Address?

Abstract Meaning Representation (AMR) is a powerful tool for representing sentence-level semantics, abstracting away from syntactic details to capture meaning in a graph format. However, its utility is limited to individual sentences, failing to capture crucial cross-sentence semantic phenomena like coreference (when different expressions refer to the same entity) and implicit roles (unspecified arguments of a predicate). This limitation hinders applications that require understanding the meaning of entire documents or longer discourses. This paper tackles this gap by proposing and developing a method to extend AMR to the multi-sentence level, creating a corpus that bridges this gap.

Methods

The core innovation of this paper is the creation of the Multi-Sentence AMR (MS-AMR) corpus. This corpus is built upon existing gold AMR annotations. The annotation process involves adding a new layer on top of these sentence-level AMRs to capture:

  • Coreference: Annotators link variables (nodes in the AMR graph) that refer to the same entity or event across different sentences. This is done by identifying clusters of variables.
  • Implicit Roles: Leveraging the PropBank predicate annotations within AMR, the system can identify expected but unfilled semantic roles. Annotators are then prompted to identify referents for these implicit roles, similar to prior work but integrated into the coreference task.
  • Bridging Relations: The corpus also annotates bridging relations, specifically part/whole and set/member relations, focusing on those between named entities and common nouns.

The annotation is performed using the Anafora toolkit, and the resulting MS-AMR representations are "stand-off" annotations, meaning they are linked to the original AMRs without modifying them, ensuring compatibility with existing AMR parsing and generation systems.

Datasets

The primary dataset introduced is the Multi-Sentence Abstract Meaning Representation (MS-AMR) corpus.

  • Source: It is annotated over previously existing gold AMRs of colloquial written English, sourced from multi-post discussion forums and web blogs.
  • Size: The corpus consists of 293 annotated documents, with an average of 27.4 AMRs (approximately 429 words) per document. This covers about 10% of the total AMR corpus.
  • Splits: A small test split is also defined, using documents from the test split of the original AMR corpus.
  • Comparison: The paper also references and compares its inter-annotator agreement scores with those from similar corpora like RED-Entities, RED-Events, and work by Gerber and Chai.

Key Insights and Ideas

  • AMR as a Foundation for Discourse: The paper demonstrates that AMR's graph-based representation is well-suited for extension to discourse-level semantics, allowing annotators to directly mark cross-sentence relations on the existing graph structure.
  • Integrated Coreference and Implicit Roles: By framing implicit role annotation within a coreference task, the MS-AMR corpus offers a more unified approach to capturing discourse phenomena, rather than treating them as separate annotation tasks.
  • Bridging Relations are Crucial: The inclusion of bridging relations highlights their importance in understanding document-level meaning, especially those connecting named entities to common nouns.
  • Annotation Challenges: Evaluating MS-AMR systems presents unique challenges, particularly in aligning variables across different AMRs when the underlying sentence-level AMRs themselves are not identical. This necessitates methods like SMATCH for variable mapping.
  • Quality Dependence: The quality of MS-AMR representations is inherently dependent on the quality of the underlying sentence-level AMRs, underscoring the importance of robust AMR parsing and annotation.

Key Results

  • Inter-Annotator Agreement (IAA):
    • Coreference: The MS-AMR corpus achieved a CoNLL-2012 F1 score of 69.86 for coreference, which is comparable to or slightly lower than traditional span-based coreference annotations, but encompasses both event and entity coreference.
  • Implicit Roles: Achieved a Cohen's Kappa of 0.59, which is lower than previous work but attributed to the broader scope and difficulty of discerning implicit roles in this context. Agreement was higher (κ=0.85) when annotators agreed that an implicit role was present.
  • Baseline System Performance: A simple baseline system combining an AMR parser, a coreference system, and an aligner achieved a 27.79 CoNLL-F1 score on the MS-AMR task. This highlights a significant gap between current system capabilities and human annotation quality.
  • SMATCH Evaluation: Document-level SMATCH scores showed that the baseline system performed significantly worse than human annotations, even when coreference information was not used, indicating the difficulty of integrating multi-sentence semantics.

Limitations and Future Work

  • Scope of Annotation: The authors acknowledge that MS-AMR does not capture all dimensions of discourse, such as rhetorical structure, temporal/aspectual structure, or information structure.
  • Domain Specificity: The corpus is derived from discussion forums and web blogs, which may introduce genre-specific challenges (e.g., "I" and "you" coreference).
  • System Performance Gap: The significant difference between baseline system performance and human annotation quality indicates a need for more advanced methods for document-level semantic representation.
  • Future Work: The paper suggests that this corpus illustrates a feasible approach for developing richer document-level representations and that the methodology could be extended to other semantic representation formalisms. The MS-AMR corpus is slated for release through the Linguistic Data Consortium.

Summary updated

BibTeX
@inproceedings{ogorman2018amr,
  author = {Tim O’Gorman and Michael Regan and Kira Griffitt and Ulf Hermjakob and Kevin Knight and Martha Palmer},
  title = {{AMR Beyond the Sentence: the Multi-sentence AMR corpus}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure