Abstract
Abstract meaning representations (AMRs) represent sentences as rooted labeled directed acyclic graphs. AMR parsing is challenging due to the lack of annotated alignments between nodes in the graphs and words in the corresponding sentences. We introduce a neural parser which treats alignments as latent variables within a joint probabilistic model of concepts, relations, and alignments. Using the variational auto-encoding framework and a continuous relaxation of discrete alignments, we show that joint modeling is preferable to traditional pipeline approaches. The parser achieves a state-of-the-art Smatch score of 74.4% on the LDC2016E25 benchmark.
Introduction
Abstract Meaning Representation (AMR) describes the semantic structure of a sentence—the "who did what to whom"—using a directed graph. However, training models to produce these graphs is difficult because the training data is "unaligned": we have the sentence and the target graph, but no explicit mapping between specific words and specific nodes. Traditionally, this was addressed using a pipeline approach: a separate tool would guess alignments, and a parser would then be trained on those guesses. This method is prone to error propagation, as the aligner is unaware of the final parsing objective.
Methodology
This paper proposes a joint model that learns to align, identify concepts, and predict relations simultaneously. The architecture treats alignments as latent variables, essentially asking which alignment makes a specific graph most likely given a sentence.
Key technical innovations include:
- Variational Autoencoding (VAE): Used to maximize a lower bound of the likelihood, as marginalizing over all possible alignments is computationally infeasible.
- Gumbel-Sinkhorn Relaxation: To make discrete alignments differentiable, the authors use a continuous relaxation that mimics a permutation matrix, allowing for end-to-end gradient descent.
- Injective Constraints: Unlike standard attention mechanisms, this model enforces that each node aligns to a specific word, forcing the model to be more decisive.
- Multi-pass Encoding: Separate BiLSTM encoders are utilized for identifying concepts and relations, allowing specialized internal representations for each task.
Experimental Results
The model was evaluated on two standard benchmarks: LDC2016E25 (R2) and LDC2015E86 (R1). The joint induction approach significantly outperformed previous models across various metrics.
As shown in the results, the parser achieved a 74.4% Smatch score on the R2 dataset, representing a 3.4% absolute improvement over the previous best result. Ablation studies confirmed that replacing latent alignment with a standard pre-trained aligner (JAMR) caused the Smatch score to drop by 0.9%, validating the benefit of joint induction.
Key Insights
- Joint Training Synergy: The parsing objective actively helps the aligner. When the model predicts a relationship between two nodes, it receives a signal that helps refine the alignment of those nodes within the sentence.
- Resolving Ambiguity: In sentences with repeated words (e.g., "long hours and lots of long nights"), a concept-only model might struggle with mapping. However, the joint relation model identifies that one "long" modifies "hours" and the other modifies "nights," guiding the alignment model to resolve the ambiguity correctly.
- Exploration via Soft Loss: Utilizing a "softer" version of the loss function (an alpha-norm) encouraged the model to explore different alignment possibilities early in training, preventing it from settling into local minima.
Conclusion and Future Work
While the joint model sets a new benchmark, it still relies on some rule-based preprocessing for "recategorization" (grouping subgraphs like dates) and simple frequency-based lookups for post-processing tasks like Wikification. Future work may involve replacing BiLSTMs with Graph Convolutional Networks (GCNs) to better capture the structural neighborhood of the AMR graph, potentially further increasing accuracy.
Summary updated