Maximum Bayes Smatch Ensemble Distillation for AMR Parsing

Young-Suk Lee, Ramón Fernandez Astudillo, Thanh Lam Hoang, Tahira Naseem, Radu Florian, Salim Roukos

Abstract

Abstract Meaning Representation (AMR) parsing has experienced an unprecedented increase in performance in the last three years, due to a mixture of effects including architecture improvements and transfer learning. Self-learning techniques have also played a role in pushing performance forward. However, for most recent high-performant parsers, the effect of self-learning and silver data augmentation seems to be fading. In this paper, we propose to overcome these diminishing returns of silver data by combining Smatch-based ensembling techniques with ensemble distillation. In an extensive experimental setup, we push single-model English parser performance to a new state-of-the-art, 85.9 (AMR 2.0) and 84.3 (AMR 3.0), and return to substantial gains from silver data augmentation. We also attain a new state-of-the-art for cross-lingual AMR parsing for Chinese, German, Italian, and Spanish. Finally, we explore the impact of the proposed technique on domain adaptation and show that it can produce gains rivaling those of human-annotated data for QALD-9 and achieve a new state-of-the-art for BioAMR.

Introduction

AMR parsing is the task of translating natural language sentences into logical graphs that capture semantic relationships—essentially determining "who is doing what to whom." While the advent of Transformer architectures and pre-trained models like BART has significantly advanced the field, recent progress has encountered a performance plateau. Traditional "self-learning"—where a model is retrained on its own machine-generated "silver" data—has begun to yield diminishing returns for top-tier models like SPRING or Structured-BART, often providing gains of less than 0.5 Smatch points.

This research addresses the plateau by suggesting that the errors in standard silver data are often the same errors the model already makes. To break through, we require higher-quality silver data that captures a consensus across diverse model architectures. We introduce Maximum Bayes Smatch Ensemble (MBSE) distillation to generate this superior training data.

Methodology: Maximum Bayes Smatch Ensemble (MBSE)

The core innovation of this work is the MBSE distillation pipeline. Rather than relying on a single model for data augmentation, we utilize an ensemble of heterogeneous parsers, such as APT (transition-based) and SPRING/Structured-BART (sequence-to-sequence).

Overview of the MBSE distillation pipeline for cross-lingual and silver data generation.

The MBSE Process

  1. Ensembling: Outputs are collected from multiple diverse parsers.
  2. Bayesian Selection: Instead of simple averaging, the system identifies the graph that maximizes the expected Smatch score relative to the distribution of all model outputs. This ensures the selected "silver" graph represents the strongest consensus among the ensemble.
  3. Distillation: This high-quality ensemble output serves as the target for training a single "student" model. This allows the student to internalize the collective wisdom of the ensemble while maintaining the inference efficiency of a single model.
  4. Cross-lingual Adaptation: For non-English languages, the pipeline utilizes Structured-mBART and translated English silver data to train target-language parsers.

Experimental Results

The MBSE distillation approach was tested across standard English benchmarks (AMR 2.0 and 3.0), specialized domains (BioAMR, QALD-9), and multiple languages.

Performance comparison on English AMR 2.0 and 3.0 benchmarks, showing the impact of MBSE silver data.

Key Findings

  • English State-of-the-Art: The method achieved new records for single-model performance with Smatch scores of 85.9 on AMR 2.0 and 84.3 on AMR 3.0. These results surpass historical human inter-annotator agreement levels (typically 79-83).
  • Cross-lingual Success: New state-of-the-art marks were established for Chinese (63.0), German (73.7), Italian (76.1), and Spanish (77.1).
  • Domain Adaptation: On the BioAMR test set, the method set a new state-of-the-art. For the QALD-9 (question answering) benchmark, the distilled silver data performed nearly as well as human-annotated data. The authors have also released the QALD-9-AMR treebank as a new resource for the community.

Discussion and Conclusion

Our results demonstrate that heterogeneity in the ensemble is crucial; combining transition-based and sequence-to-sequence models provides a broader range of candidate graphs, leading to a more robust consensus. Furthermore, by optimizing for the Smatch metric during the data generation phase, we align training targets with final evaluation criteria.

Despite these gains, challenges remain. The computational cost of generating MBSE silver data is high due to the NP-complete nature of Smatch calculations. Additionally, specialized domains still struggle with "Out of Vocabulary" (OOV) entities, and cross-lingual models face difficulties with language-specific features like German compound negations. Future research will focus on improving the efficiency of graph consensus calculations and developing better tokenization strategies for complex terminology.

Summary updated

Abstract

AMR parsing has experienced an unprecendented increase in performance in the last three years, due to a mixture of effects including architecture improvements and transfer learning. Self-learning techniques have also played a role in pushing performance forward. However, for most recent high performant parsers, the effect of self-learning and silver data augmentation seems to be fading. In this paper we propose to overcome this diminishing returns of silver data by combining Smatch-based ensembling techniques with ensemble distillation. In an extensive experimental setup, we push single model English parser performance to a new state-of-the-art, 85.9 (AMR2.0) and 84.3 (AMR3.0), and return to substantial gains from silver data augmentation. We also attain a new state-of-the-art for cross-lingual AMR parsing for Chinese, German, Italian and Spanish. Finally we explore the impact of the proposed technique on domain adaptation, and show that it can produce gains rivaling those of human annotated data for QALD-9 and achieve a new state-of-the-art for BioAMR.
BibTeX
@misc{lee2021maximum,
  author = {Young-Suk Lee and Ramón Fernandez Astudillo and Thanh Lam Hoang and Tahira Naseem and Radu Florian and Salim Roukos},
  title = {{Maximum Bayes Smatch Ensemble Distillation for AMR Parsing}},
  year = {2021},
  eprint = {2112.07790},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.07790},
}
Figure