Leveraging Denoised Abstract Meaning Representation for Grammatical Error Correction

Hejing Cao, Dongyan Zhao

Introduction

Grammatical Error Correction (GEC) is a fundamental natural language processing task that aims to transform ungrammatical sentences into fluent, correct versions. Current state-of-the-art approaches typically fall into two categories: data-enhanced models and knowledge-enhanced models. Data-enhanced models rely on massive synthetic datasets, often generated via back-translation, which are computationally expensive and slow to train. Knowledge-enhanced models utilize human-designed rules, which are difficult to scale and require deep linguistic expertise.

Researchers from Peking University have proposed a novel middle ground: AMR-GEC. This model is the first to incorporate Abstract Meaning Representation (AMR) as external semantic knowledge for GEC. AMR provides a rooted, labeled graph that captures the core semantic structure of a sentence—"who is doing what to whom"—independent of its surface syntax. By leveraging this semantic skeleton, AMR-GEC can bypass surface-level noise and focus on the underlying meaning.

The AMR-GEC Framework

The core challenge in using AMR for GEC is that AMRs generated from "errorful" sentences are often noisy or unreliable. To address this, the authors developed a sequence-to-sequence architecture with three primary components:

  1. Semantic Aggregated Encoder: This component fuses a standard Transformer sequence encoder with a Graph Neural Network (GNN) encoder. It processes a hybrid "Sequence-AMR graph" that combines linear word order with semantic edges.
  2. Denoising Function: To ensure robustness against flawed AMRs, the model employs a denoising strategy during training. This involves masking nodes, edges, or entire subgraphs (replacing them with a [mask] label), forcing the model to learn which semantic structures are reliable.
  3. Graph Construction: The system utilizes the SPRING parser for AMR generation and the JAMR alignment tool to map graph nodes back to specific sentence tokens.
Architecture of the AMR-GEC model featuring denoising and semantic aggregation.

Experimental Setup

The model was evaluated on two benchmark datasets: BEA-2019 (Write & Improve and LOCNESS corpora) and CoNLL-2014 (expert-annotated English sentences). The researchers used a T5 model as their baseline and compared AMR-GEC against high-performing systems like GECToR and SynGEC.

Results and Analysis

The inclusion of semantic knowledge proved highly effective. AMR-GEC achieved an $F_{0.5}$ score of 68.4 on the BEA-2019 test set, outperforming several strong baselines that rely on synthetic data. Notably, the model reduced training time by 32% compared to traditional data-heavy approaches while maintaining comparable inference speeds.

Semantics as a Shortcut

One of the key insights is that AMR helps the model ignore "surface noise" such as tense errors or misplaced prepositions. As shown in the performance breakdown by error type, AMR-GEC significantly outperforms the T5 baseline in categories like punctuation (PUNCT) and prepositions (PREP). These errors often do not alter the underlying AMR structure, allowing the semantic graph to act as a reliable guide for correction.

Performance comparison between T5-GEC and AMR-GEC across different error types.

Handling Long-Distance Dependencies

Standard sequence models often struggle with dependencies across long spans. Because AMR explicitly connects related concepts with edges regardless of their textual distance, the graph encoder enables the model to maintain consistency more effectively over long sentences.

The Role of Denoising

Ablation studies highlighted the necessity of the denoising strategy. Without it, the model tended to "hallucinate" corrections based on flawed graph structures. Implementing node/edge level denoising increased the $F_{0.5}$ score by 1.57 points over a version using raw, noisy AMRs.

Conclusion and Future Work

AMR-GEC demonstrates that semantic graphs can serve as a powerful and efficient source of knowledge for GEC. While the model is currently limited by the quality of off-the-shelf AMR parsers, its performance is expected to improve alongside advancements in AMR parsing technology. Future work will explore more sophisticated ensemble strategies and the integration of "Right-to-Left" (R2L) reranking to further boost accuracy.

Summary updated

Abstract

Grammatical Error Correction (GEC) is the task of correcting errorful sentences into grammatically correct, semantically consistent, and coherent sentences. Popular GEC models either use large-scale synthetic corpora or use a large number of human-designed rules. The former is costly to train, while the latter requires quite a lot of human expertise. In recent years, AMR, a semantic representation framework, has been widely used by many natural language tasks due to its completeness and flexibility. A non-negligible concern is that AMRs of grammatically incorrect sentences may not be exactly reliable. In this paper, we propose the AMR-GEC, a seq-to-seq model that incorporates denoised AMR as additional knowledge. Specifically, We design a semantic aggregated GEC model and explore denoising methods to get AMRs more reliable. Experiments on the BEA-2019 shared task and the CoNLL-2014 shared task have shown that AMR-GEC performs comparably to a set of strong baselines with a large number of synthetic data. Compared with the T5 model with synthetic data, AMR-GEC can reduce the training time by 32\% while inference time is comparable. To the best of our knowledge, we are the first to incorporate AMR for grammatical error correction.
BibTeX
@inproceedings{cao2023leveraging,
  author = {Hejing Cao and Dongyan Zhao},
  title = {{Leveraging Denoised Abstract Meaning Representation for Grammatical Error Correction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.449},
}
Figure