Structural Neural Encoders for AMR-to-text Generation

Marco Damonte, Shay B. Cohen

Abstract

Abstract Meaning Representation (AMR) to natural language generation is a complex task due to the non-linear, graph-based nature of semantic representations. This paper explores the impact of explicitly modeling graph structures, particularly "reentrancies" (nodes with multiple parents), using Graph Convolutional Networks (GCNs). We demonstrate that a "Sequence on Top of Structure" architecture—where a GCN refines embeddings before a BiLSTM encoder—achieves state-of-the-art results on standard benchmarks, particularly for sentences with high structural complexity.

Introduction: The Challenge of AMR Graphs

AMR-to-text generation involves transforming a semantic graph—representing "who did what to whom"—into a coherent natural language sentence. Unlike traditional syntactic trees, AMRs are directed acyclic graphs that frequently contain reentrancies. A reentrancy occurs when a single concept serves multiple roles; for example, in "He eats pizza with his fingers," the concept "he" is both the agent of eating and the owner of the fingers.

Comparison of AMR representations: (a) semantic graph with reentrancy, (b) linearized string, (c) tree structure, and (d) graph structure.

Historically, models simplified this task by linearizing the graph into a sequence or a tree, effectively discarding these multi-parent links. This research investigates whether preserving the full graph structure improves generation and identifies the optimal neural architecture for exploiting such structural information.

Methodology

The authors evaluate three distinct encoder architectures to determine the value of structural awareness:

  1. Sequential Encoder: A standard BiLSTM treating the linearized AMR as a string.
  2. Tree-structured Encoder: A bidirectional TreeLSTM that captures hierarchy but ignores reentrancies by duplicating shared nodes.
  3. Graph-structured Encoder: A Graph Convolutional Network (GCN) that propagates information across all edges, including reentrancies, allowing nodes to aggregate context from multiple parents.

The Stacking Strategy

A key contribution of this work is the discovery of the optimal "stacking" order for hybrid models. The researchers tested two configurations:

  • Structure on Top of Sequence: BiLSTM → GCN/TreeLSTM.
  • Sequence on Top of Structure (GCNSEQ): Initial embeddings are first "refined" by a GCN based on the graph topology, and these structurally-enriched embeddings are then processed by a BiLSTM.

To integrate edge labels (like :arg0 or :instrument) into the GCN, the AMR is converted into a "Levi graph," where edges are treated as nodes. This transformation enables the model to learn unlabeled connections between concepts and their specific relations.

Experimental Setup and Results

The models were benchmarked on two standard LDC datasets: LDC2015E86 and the larger LDC2017T10. Performance was measured using BLEU and Meteor scores.

The GCNSEQ model established new state-of-the-art results:

  • LDC2015E86: 24.40 BLEU (a 1.1-point improvement over previous SOTA).
  • LDC2017T10: 24.54 BLEU (a 1.24-point improvement).

Crucially, the advantage of the graph-structured approach scales with complexity. In an analysis of sentences containing 6–20 reentrancies, the GCN model outperformed the sequential baseline by 3.08 Meteor points, proving its superior ability to handle long-range dependencies and co-reference.

Discussion and Key Insights

  • Reentrancies as Structural Anchors: The performance gap widens as structural complexity increases. Graph encoders excel at generating correct pronouns and managing control structures (e.g., "X persuaded Y to do Z") because they maintain explicit links between shared concepts.
  • Shortcutting Semantic Distance: In a linear string, related concepts may be separated by dozens of tokens. In a graph, they are adjacent. GCNs allow the model to capture these dependencies directly, bypassing the "forgetting" issues common in RNNs.
  • Structural Pre-processing: The success of the GCN → BiLSTM pipeline suggests that structural context serves as a powerful pre-processing step, providing the sequence model with a more informed starting point for encoding.

Limitations and Future Work

Despite significant gains, challenges remain. AMR graphs often lack explicit tense information (e.g., "eat-01" could be "ate" or "will eat"), leading to occasional verb form errors. Additionally, while co-reference handling is improved, the model still occasionally falters on pronoun gender or type. Future research may involve training objectives specifically tailored for control verbs and co-reference resolution beyond standard cross-entropy loss.

Summary updated

BibTeX
@inproceedings{damonte2019structural,
  author = {Marco Damonte and Shay B. Cohen},
  title = {{Structural Neural Encoders for AMR-to-text Generation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/n19-1366},
}
Figure