Abstract
Grammatical error correction (GEC) is the task of detecting and correcting grammatical errors in texts written by second language learners. The statistical machine translation (SMT) approach to GEC, in which sentences written by second language learners are translated to grammatically correct sentences, has achieved state-of-the-art accuracy. However, the SMT approach is unable to utilize global context. In this paper, we propose a novel approach to improve the accuracy of GEC, by exploiting the n-best hypotheses generated by an SMT approach. Specifically, we build a classifier to score the edits in the n-best hypotheses. The classifier can be used to select appropriate edits or re-rank the n-best hypotheses. We apply these methods to a state-of-the-art GEC system that uses the SMT approach. Our experiments show that our methods achieve statistically significant improvements in accuracy over the best published results on a benchmark test dataset on GEC.
Introduction
Statistical Machine Translation (SMT) has emerged as a powerhouse for Grammatical Error Correction (GEC). By treating the correction process as a "translation" from ungrammatical to grammatical English, SMT systems have achieved impressive results. However, most SMT systems are phrase-based, meaning they operate on small windows of text. This local focus often causes them to miss the "big picture," particularly long-distance dependencies like subject-verb agreement across long prepositional phrases.
To address this limitation, this research introduces a method to bridge the gap between local phrase-based translation and global sentence context. By training a classifier to evaluate the "n-best" alternative translations produced by an SMT system, we can leverage linguistic features and global context to either rerank these alternatives or surgically select the best edits from across multiple hypotheses.
Methodology
The proposed approach consists of two stages. First, a standard SMT system (Moses) generates a list of the top n possible corrections (the n-best hypotheses). Second, a novel Edit Classifier is applied to refine these results.
The Edit Classifier
A binary classifier was developed using Confidence-Weighted learning to determine the validity of specific "edits" suggested by the SMT system. An edit is defined as a change from the source sentence to a hypothesis. The classifier utilizes four categories of features to capture both local and global information:
- SMT Features: The rank of the hypothesis containing the edit.
- Lexical & POS Features: The words involved and their Parts-of-Speech.
- Context Features: Information about neighboring words and the "heads" of the nearest Noun Phrases (NP) or Verb Phrases (VP), identified via constituency parsing. This allows the system to capture long-range dependencies.
- Language Model Features: Scores from a 5-gram language model trained on Wikipedia to assess the naturalness of the proposed edit.
Strategies for Improvement
Two primary strategies were explored for utilizing the classifier's scores:
- Reranking: The original n-best list is re-ordered by incorporating the classifier's average edit scores as a new feature, followed by Minimum Error Rate Training (MERT) to optimize the final selection.
- Edit Selection: This approach looks across the top 5 hypotheses. It greedily selects the highest-scoring, non-overlapping valid edits and applies them to the original source sentence. This can create a "Frankenstein" hypothesis that is superior to any single output originally generated by the SMT system.
Experimental Setup
The system was trained and evaluated using standard GEC benchmarks:
- NUCLE (NUS Corpus of Learner English): Primary training data for the translation model and edit classifier.
- Lang-8 Corpus: A large-scale learner dataset used to augment the SMT training data.
- CoNLL-2013 & CoNLL-2014: Used as development and test sets, respectively.
- English Wikipedia: A corpus of ~1.78 billion words used to train the 5-gram language model.
Results and Discussion
Experiments demonstrate that exploiting n-best hypotheses significantly improves GEC performance. While the baseline SMT system was already strong, the proposed methods pushed the accuracy further.
As shown in the results table, the baseline system (F0.5 of 40.58) already outperformed existing state-of-the-art systems. The Edit Selection method (using the top 5 hypotheses) further improved this to an F0.5 score of 41.19. This improvement was statistically significant and outperformed other top-performing systems from the CoNLL-2014 shared task, such as CAMB and CUUI.
Key insights from the results include:
- N-Best Richness: Valuable corrections often exist outside the top-ranked hypothesis.
- Global Context: Using NP and VP heads helps resolve complex errors like subject-verb agreement that phrase-based SMT typically misses.
- Edit Selection Superiority: The fact that Edit Selection outperformed Reranking suggests that the optimal correction is often distributed across multiple SMT guesses.
Conclusion and Future Work
This work demonstrates that an edit classifier leveraging global context can significantly enhance phrase-based SMT for GEC. However, challenges remain. The initial SMT search (pruning) might discard correct edits before the classifier can see them. Additionally, the use of constituency parsers introduces computational overhead. Future work could explore using this classifier to ensemble outputs from diverse GEC systems, potentially combining the strengths of SMT with rule-based or neural approaches.
Summary updated