Abstract
We propose a novel language-independent approach to improve the efficiency for Grammatical Error Correction (GEC) by dividing the task into two subtasks: Erroneous Span Detection (ESD) and Erroneous Span Correction (ESC). ESD identifies grammatically incorrect text spans with an efficient sequence tagging model. Then, ESC leverages a seq2seq model to take the sentence with annotated erroneous spans as input and only outputs the corrected text for these spans. Experiments show our approach performs comparably to conventional seq2seq approaches in both English and Chinese GEC benchmarks with less than 50% time cost for inference.
Introduction
Standard Grammatical Error Correction (GEC) systems primarily rely on sequence-to-sequence (seq2seq) models. While these models achieve high accuracy, they suffer from significant inference latency. This bottleneck arises because seq2seq models treat GEC as a translation task, generating the entire corrected sentence token-by-token. In practice, most tokens in an erroneous sentence are correct; thus, standard models waste computational resources 're-typing' or copying correct text from the source to the target. This paper addresses this 'copying bottleneck' by proposing a framework that focuses computation exclusively on spans requiring correction.
Methodology
The proposed framework employs a two-stage 'detect-then-edit' pipeline that is inherently language-independent.
Erroneous Span Detection (ESD)
The first stage utilizes an efficient sequence tagging model (e.g., RoBERTa or BERT) to scan the input sentence. The model assigns a binary tag to each token: '0' for correct tokens and '1' for tokens within an erroneous span. This stage acts as a filter, allowing the system to identify exactly which parts of the sentence need modification.
Erroneous Span Correction (ESC)
The input sentence, augmented with span tags (e.g., <s1> ... </s1>), is passed to a specialized seq2seq model. Unlike traditional decoders, the ESC decoder is trained to output only the corrected text for the identified spans, significantly reducing the number of decoding steps.
To ensure robustness against imperfect detection, the ESC model is trained with random span annotations, similar to the masking strategy used in SpanBERT. This enables the model to handle errors in the ESD stage gracefully.
Experimental Setup
The models were evaluated on major English (CoNLL-2014, BEA-2019) and Chinese (NLPCC-2018) GEC benchmarks. Training data included diverse sources such as FCE, Lang-8, NUCLE, and W&I+LOCNESS. To enhance performance, the authors pretrained both components on a massive synthetic dataset of 260 million sentence pairs generated via back-translation and sentence corruption.
Results and Discussion
The 'detect-then-edit' approach demonstrates that GEC is a sparse task where focusing on 'noise' (errors) rather than 'signal' (correct text) yields massive efficiency gains.
Efficiency and Speed
The proposed method reduces inference time by more than 50% compared to standard Transformer-based seq2seq baselines. This speedup is consistent across various batch sizes and is primarily driven by the reduction in decoding steps—for instance, requiring only ~7,600 steps compared to over 21,000 in a standard test scenario.
Accuracy and Performance
Despite the focus on efficiency, the model maintains competitive accuracy. On the BEA-19 English test set, it achieved an $F_{0.5}$ score of 66.9, outperforming several complex ensemble and rescoring methods. The system's success in both English and Chinese benchmarks underscores its language-independent nature. Furthermore, because the ESD stage uses a probability threshold, the system can be tuned for different precision/recall requirements.
Conclusion
This research presents a highly efficient framework for GEC that significantly mitigates the computational overhead of seq2seq models. While highly effective for local span corrections, future work may address extremely complex errors requiring global restructuring. The 'detect-then-rewrite' philosophy holds promise for other text-editing tasks such as simplification and style transfer.
Summary updated