Abstract
Predicting linearized Abstract Meaning Representation (AMR) graphs using pre-trained sequence-to-sequence Transformer models has recently led to large improvements on AMR parsing benchmarks. These parsers are simple and avoid explicit modeling of structure but lack desirable properties such as graph well-formedness guarantees or built-in graph-sentence alignments. In this work we explore the integration of general pre-trained sequence-to-sequence language models and a structure-aware transition-based approach. We depart from a pointer-based transition system and propose a simplified transition set, designed to better exploit pre-trained language models for structured fine-tuning. We also explore modeling the parser state within the pre-trained encoder-decoder architecture and different vocabulary strategies for the same purpose. We provide a detailed comparison with recent progress in AMR parsing and show that the proposed parser retains the desirable properties of previous transition-based approaches, while being simpler and reaching the new parsing state of the art for AMR 2.0, without the need for graph re-categorization.
Introduction
Abstract Meaning Representation (AMR) parsing involves translating natural language sentences into rooted, directed acyclic graphs that capture semantic relationships. Recent advancements have seen a divergence between two primary methodologies: Linearization Parsers, which treat graph generation as a sequence-to-sequence translation task, and Transition-based Parsers, which construct graphs incrementally through a series of discrete actions.
While linearization models benefit from the generative power of pre-trained language models like BART, they often struggle with structural validity and lose explicit alignments between words and nodes. Conversely, transition-based models offer structural guarantees and clear alignments but frequently rely on complex preprocessing (re-categorization) and external linguistic tools. This paper introduces StructBART, a model designed to synthesize the strengths of both approaches by integrating a simplified transition system directly into the BART architecture.
Methods
StructBART modifies the standard BART architecture to support structured fine-tuning through three primary innovations:
- Simplified Transition System: The authors reduced the transition set to six core actions:
SHIFT,<string>(node creation),COPY,LA/RA(edge creation), andROOT. By using node names as actions, the model leverages BART’s existing vocabulary. - Pointer Network for Edges: To establish edges between nodes, the model repurposes a decoder self-attention head. This head acts as a pointer, identifying previous steps in the action history to connect nodes.
- Hard Attention for Alignments: A "token cursor" mechanism forces the model to attend only to specific input tokens when generating corresponding nodes, effectively internalizing word-to-node alignments.
Experimental Setup
The model was evaluated on standard benchmarks including AMR 1.0, 2.0, and 3.0. The authors also explored the impact of "silver" data—synthetic sentences from PropBank and SQuAD 2.0—to augment the training set. A key focus of the experiments was the removal of graph re-categorization, a common but cumbersome preprocessing step in AMR parsing.
Results and Discussion
StructBART achieved a new state-of-the-art SMATCH score of 84.7 on AMR 2.0, outperforming previous models even those utilizing significantly more external data. Fine-grained analysis revealed that the model is particularly effective at capturing complex graph topologies, such as re-entrancies and Semantic Role Labeling (SRL) structures.
Ablation studies confirmed that the pre-trained BART encoder is the primary driver of performance, contributing a ~10-point gain over randomly initialized models. Furthermore, the results indicate that sharing a joint vocabulary between the language model and the parser is as effective as specialized vocabularies, simplifying the training pipeline.
Conclusion
StructBART demonstrates that integrating structural constraints into pre-trained sequence-to-sequence models yields superior results to pure linearization. By simplifying the transition system and utilizing internal attention mechanisms for pointers and alignments, the authors provide a robust framework that achieves state-of-the-art performance without the need for complex graph preprocessing. Future work may extend these principles to other structured prediction tasks such as SQL parsing or code generation.
Summary updated