Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction

Mengyun Chen, Tao Ge, Xingxing Zhang, Furu Wei, Ming Zhou

Abstract

Standard Grammatical Error Correction (GEC) models, typically based on autoregressive sequence-to-sequence (seq2seq) architectures, suffer from significant efficiency bottlenecks due to redundant token generation. Most source sentences are largely correct, yet models spend considerable computation copying correct text. We present a two-stage approach: Erroneous Span Detection (ESD) and Erroneous Span Correction (ESC). By identifying specific error spans and generating corrections only for those segments, our method reduces inference time by over 50% while maintaining or exceeding state-of-the-art accuracy on standard benchmarks.

Introduction

The current gold standard for GEC involves seq2seq models, usually Transformer-based. However, these models are autoregressive, generating the entire output token-by-token. In practice, the vast majority of a source sentence is grammatically correct. The model's "thinking time" is largely wasted on copying correct text from source to target. This makes real-time deployment in applications like word processors or chat apps computationally expensive and slow.

Methodology: The Detect-then-Edit Pipeline

We propose a pipeline that decouples error detection from correction, utilizing two specialized components:

Erroneous Span Detection (ESD)

Instead of a heavy seq2seq model, we employ an efficient sequence tagging model (fine-tuned from RoBERTa or BERT). This model performs binary classification on every token: 0 for correct and 1 for incorrect. This stage identifies the specific "spans" requiring modification.

Erroneous Span Correction (ESC)

The ESC is a Transformer-big seq2seq model that receives the original sentence with detected spans marked by special tags (e.g., <s1> erroneous text </s1>). Crucially, the decoder is trained to output only the corrected text for these specific spans, bypassing the need to rewrite the entire sentence.

The two-stage GEC architecture involving Erroneous Span Detection (ESD) and Correction (ESC).

To ensure robustness against imperfect detections, we utilize a training strategy inspired by SpanBERT. During training, we randomly select spans to "correct," teaching the model to handle various boundaries and even validate spans that may already be correct.

Experimental Setup and Datasets

The approach was evaluated on English and Chinese benchmarks:

  • English: BEA-2019 and CoNLL-2014 test sets, with training data from FCE, Lang-8, NUCLE, and W&I+LOCNESS.
  • Chinese: NLPCC-2018 shared task dataset.
  • Pre-training: We synthesized a dataset of 260 million sentence pairs using back-translation and sentence corruption to achieve state-of-the-art performance.

Results and Analysis

Efficiency Gains

The "copying" bottleneck was significantly reduced. For 872 incorrect sentences, a standard model required 21,065 decoding steps, whereas our ESC model required only 7,647—a nearly 3x reduction in the most expensive part of the pipeline. Overall, this translates to a 50% reduction in total inference time across various batch sizes.

Performance Comparison

Despite the speedup, the model remains highly competitive. On the BEA-19 benchmark, our pre-trained model achieved an $F_{0.5}$ score of 66.9, outperforming the standard seq2seq baseline of 65.5. The ESD model also acts as a high-speed gatekeeper; if no errors are detected, the system bypasses the heavy ESC model entirely, saving massive amounts of compute on clean text.

Performance comparison of the proposed approach against state-of-the-art models on standard benchmarks.

Controllability

The ESD model outputs probabilities, allowing for an adjustable detection threshold. Users can tune the system to be "conservative" (higher precision) or "aggressive" (higher recall), a level of control difficult to achieve with standard black-box seq2seq models.

Conclusion

Our two-stage GEC framework addresses the efficiency bottleneck of traditional models by focusing computation only on erroneous spans. This approach is language-independent and offers a significant speedup without sacrificing accuracy. Future work includes exploring model distillation for the ESC component and applying span-specific decoding to other text-rewriting tasks like simplification or style transfer.

Summary updated

Abstract

We propose a novel language-independent approach to improve the efficiency for Grammatical Error Correction (GEC) by dividing the task into two subtasks: Erroneous Span Detection (ESD) and Erroneous Span Correction (ESC). ESD identifies grammatically incorrect text spans with an efficient sequence tagging model. Then, ESC leverages a seq2seq model to take the sentence with annotated erroneous spans as input and only outputs the corrected text for these spans. Experiments show our approach performs comparably to conventional seq2seq approaches in both English and Chinese GEC benchmarks with less than 50% time cost for inference.
BibTeX
@inproceedings{chen2020improving,
  author = {Mengyun Chen and Tao Ge and Xingxing Zhang and Furu Wei and Ming Zhou},
  title = {{Improving the Efficiency of Grammatical Error Correction with Erroneous Span Detection and Correction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=AQxQRnLMJPN},
}
Figure