System Combination for Grammatical Error Correction

Raymond Hendy Susanto, Peter Phandi, Hwee Tou NgEMNLP 2014 · OpenReview

Abstract

Raymond Hendy Susanto, Peter Phandi, Hwee Tou Ng. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2014.

Introduction

Grammatical Error Correction (GEC) is a complex task due to the diversity of error types and their interdependencies. Traditionally, two paradigms have dominated the field: classification-based approaches and Statistical Machine Translation (SMT). Classification-based systems utilize specialized modules for specific error types, offering precision but often struggling with broader linguistic context and word interactions. Conversely, SMT treats GEC as a translation task from "incorrect" to "correct" English, naturally handling word interactions but requiring vast amounts of annotated data. This paper explores a hybrid approach, demonstrating that combining these two paradigms significantly improves performance.

Methods

The researchers developed four distinct component systems to be merged:

  1. Classification Pipelines (P1 & P2): These systems process text through a sequence of error-specific modules. P1 and P2 differ only in the ordering of "Noun Number" and "Article" corrections, a variation found to produce complementary outputs.
  2. SMT Systems (S1 & S2): Two phrase-based SMT systems were trained. S1 utilized separate phrase tables for different datasets, while S2 employed a single concatenated table with an additional word-level Levenshtein distance feature.
The sequential steps for the two classification-based pipelines, P1 and P2.

The core innovation lies in the Multi-Engine Machine Translation (MEMT) system combination. Rather than selecting the single best sentence output, MEMT aligns words across all system outputs using the METEOR matcher. It then constructs a "confusion network" and uses beam search to find the optimal path, effectively "stitching together" the most accurate corrections from each individual system.

Datasets

The systems were trained and evaluated using standard GEC benchmarks and large-scale corpora:

  • NUCLE: Essays from students at the National University of Singapore, used for training.
  • Lang-8: A large-scale corpus of learner English mined from social media.
  • CoNLL-2013 & 2014: Used for development/tuning and final testing, respectively.
  • English Wikipedia: Used to train a massive 5-gram language model (nearly 1.8 billion tokens) to ensure fluency in the corrected output.

Results and Discussion

The combined system achieved a state-of-the-art $F_{0.5}$ score of 39.39% on the CoNLL-2014 test set, outperforming the previous best system (CAMB, 37.33%).

Performance comparison of individual pipelines, SMT systems, combined systems, and the top-performing CoNLL-2014 shared task entries.

Key insights from the evaluation include:

  • Complementary Strengths: Classification pipelines generally achieved higher recall (finding more errors), while SMT systems provided higher precision (making fewer mistakes). The combination leveraged both, resulting in a system that was both bold and accurate.
  • Alignment Efficacy: The use of METEOR for word-level alignment was critical. By recognizing stems and synonyms, the system could merge corrections from different paradigms (e.g., a noun-number fix from a pipeline and a preposition fix from SMT) without creating ungrammatical word forms.
  • Optimal System Count: Further experiments revealed that performance peaks when combining the top 5 diverse systems. Adding lower-quality systems eventually degrades the overall output, suggesting that diversity and quality are more important than sheer quantity in system combination.

Conclusion

This work demonstrates that system combination via word-level alignment is a highly effective strategy for GEC. By merging the precision of rule-based pipelines with the contextual awareness of SMT, the authors established a new performance benchmark. Future work will focus on reducing the labor-intensive nature of building individual classifiers and improving alignment techniques to prevent the generation of ungrammatical "franken-sentences" when systems disagree significantly.

Summary updated

BibTeX
@inproceedings{susanto2014system,
  author = {Raymond Hendy Susanto and Peter Phandi and Hwee Tou Ng},
  title = {{System Combination for Grammatical Error Correction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2014},
  url = {https://openreview.net/forum?id=jClM0bTVx58},
}
Figure