Exploiting N-Best Hypotheses to Improve an SMT Approach to Grammatical Error Correction

Duc Tam Hoang, Shamil Chollampatt, Hwee Tou Ng

Statistical Machine Translation (SMT) has long been a foundational approach for Grammatical Error Correction (GEC), treating the correction process as a translation task from "erroneous English" to "fluent English." However, standard phrase-based SMT often suffers from a "local vision" problem, where it excels at fixing short-range errors but fails to maintain global grammatical consistency, such as long-distance subject-verb agreement.

This paper introduces a novel two-stage approach to address these limitations by moving beyond the standard practice of selecting a single "best" translation. Instead, the authors propose an "edit selection" method that synthesizes a superior correction by combining the most reliable individual edits from across multiple candidate hypotheses.

Methodology: Beyond the 1-Best Hypothesis

The core of the proposed system is a two-stage pipeline designed to inject global context into the SMT framework.

1. The Edit Classifier

The system first extracts every "edit" (the specific change from the source sentence to a candidate correction) from the SMT system's $n$-best hypotheses. A binary classifier is then trained to distinguish between valid corrections and erroneous changes. Unlike the base SMT system, this classifier leverages features that capture broader linguistic context:

  • Syntactic Context: Utilizing a parser to identify the nearest noun phrase (NP) or verb phrase (VP) heads to ensure grammatical coherence (e.g., checking if a verb change aligns with its distant subject).
  • Language Model Scores: Evaluating the fluency of the sentence before and after an edit using a massive 5-gram model trained on Wikipedia.
  • Lexical and POS Features: Analyzing specific word choices and their parts of speech to identify common error patterns.
Table of features used by the edit classifier, including lexical, POS, context, and language model features.

2. Exploiting the N-Best List

With a trained edit classifier, the authors propose two strategies for improving the final output:

  • Reranking: The classifier's scores are used as a new feature to re-sort the $n$-best list, allowing the system to pick a better overall sentence than the original SMT decoder.
  • Edit Selection: This novel approach identifies all "valid" edits across the top $n$ hypotheses. It then greedily selects the highest-scoring, non-overlapping edits and applies them to the original source sentence. This allows the system to "Frankenstein" a new sentence that may be more accurate than any single hypothesis produced by the decoder.

Experimental Results

The system was evaluated on the CoNLL-2014 shared task benchmark, a standard for GEC performance. The results demonstrate that both reranking and edit selection provide significant gains over the baseline SMT system.

Performance comparison ($F_{0.5}$) between the baseline SMT system, reranking, and the proposed edit selection method on the CoNLL-2014 test set.

As shown in the results, the Edit Selection (5-best) method achieved the highest $F_{0.5}$ score of 41.19, a substantial improvement over the 40.58 baseline. For comparison, the top-performing system in the original CoNLL-2014 shared task (CAMB) reached an $F_{0.5}$ of 37.33. The primary driver of this improvement was increased precision—the system became significantly better at avoiding "bad" corrections that would introduce new errors.

Insights and Future Directions

A key takeaway from the ablation studies is that "Context Features" and "Language Model Features" are the most critical contributors to the classifier's success. This confirms that the primary weakness of traditional SMT in GEC is indeed its lack of global structural awareness.

While the system is currently limited by the SMT decoder's ability to generate good edits in its $n$-best list, the "edit selection" framework is inherently general. Future work could involve combining hypotheses from diverse GEC architectures—such as rule-based, neural, and statistical systems—to leverage the unique strengths of each approach within a single, unified correction framework.

Summary updated

Abstract

Grammatical error correction (GEC) is the task of detecting and correcting grammatical errors in texts written by second language learners. The statistical machine translation (SMT) approach to GEC, in which sentences written by second language learners are translated to grammatically correct sentences, has achieved state-of-the-art accuracy. However, the SMT approach is unable to utilize global context. In this paper, we propose a novel approach to improve the accuracy of GEC, by exploiting the n-best hypotheses generated by an SMT approach. Specifically, we build a classifier to score the edits in the n-best hypotheses. The classifier can be used to select appropriate edits or re-rank the n-best hypotheses. We apply these methods to a state-of-the-art GEC system that uses the SMT approach. Our experiments show that our methods achieve statistically significant improvements in accuracy over the best published results on a benchmark test dataset on GEC.
BibTeX
@inproceedings{hoang2016exploiting,
  author = {Duc Tam Hoang and Shamil Chollampatt and Hwee Tou Ng},
  title = {{Exploiting N-Best Hypotheses to Improve an SMT Approach to Grammatical Error Correction}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure