Abstract
This paper proposes a novel approach to the problem of training classifiers to detect and correct grammar and usage errors in text by selectively introducing mistakes into the training data. When training a classifier, we would like the distribution of examples seen in training to be as similar as possible to the one seen in testing. In error correction problems, such as correcting mistakes made by second language learners, a system is generally trained on correct data, since annotating data for training is expensive. Error generation methods avoid expensive data annotation and create training data that resemble non-native data with errors.
We apply error generation methods and train classifiers for detecting and correcting article errors in essays written by non-native English speakers; we show that training on data that contain errors produces higher accuracy when compared to a system that is trained on clean native data. We propose several training paradigms with error generation and show that each such paradigm is superior to training a classifier on native data. We also show that the most successful error generation methods are those that use knowledge about the article distribution and error patterns observed in non-native text.
Introduction
Most grammar correction systems are traditionally trained as selection tasks: given a context, the model chooses the most likely word (e.g., "a," "the," or nothing). However, real-world proofreading is a correction task. In correction, the system already has the writer's original choice (the "source"), which it must weigh against its own predictions.
The challenge is that ESL (English as a Second Language) learners are correct about 90% of the time. If a model is trained only on clean native text, it never learns how to weigh its own confidence against the writer's original choice. To a model trained on "perfect" English, every sentence written by a learner looks like a potential error, leading to "over-correction" and lower accuracy. The authors aim to bridge this gap by creating training data that looks like real learner writing—errors and all.
Methodology: Training with Errors (TWE)
The authors focus on article errors (a, an, the, or no article). They use a discriminative learning approach (Averaged Perceptron) and compare a baseline trained on clean data (TrainClean) against several "Train With Errors" (TWE) paradigms:
- TWE (General): Articles in the training data are replaced with a different article at random with a fixed probability.
- TWE (ArtDistr): Changes the distribution of articles in the training set to match the frequency seen in ESL writing (either before or after native annotation).
- TWE (ErrorDistr): The most sophisticated method. It uses specific error statistics from annotated ESL corpora. For example, if data shows that Russian speakers omit a required "the" 14% of the time, the method purposefully deletes "the" in 14% of the training examples.
Crucially, these TWE models can use the "source" word (the potentially wrong article) as a feature, whereas the TrainClean model cannot, because in its world, the source is always correct.
Datasets and Evaluation
- Training Data: 2 million examples extracted from English Wikipedia (native, clean text).
- ESL Evaluation Data:
- Chinese: From the CLEC corpus (students of all levels).
- Czech & Russian: From the ICLE corpus (advanced learners).
- Annotation: A portion of the ESL data was manually error-tagged by native speakers to provide a "gold standard" for evaluation and to generate error statistics for the TWE methods.
Key Results
The TWE approach consistently outperformed the TrainClean approach across all language groups. In all cases, training on "corrupted" data led to higher accuracy on real-world ESL essays than training on "perfect" Wikipedia text.
- Russian Speakers: The best TWE model achieved a 16.05% error reduction over the baseline.
- Czech Speakers: Achieved a 14.69% error reduction.
- Chinese Speakers: Achieved a 10.06% error reduction.
The results also showed that while random error injection (General) helps, using specific error distributions (ErrorDistr) is generally superior, particularly for learners whose native languages (L1) lack an article system, such as Russian and Czech.
Conclusion
This paper demonstrates that when training AI to fix grammar, "perfect" training data is actually a problem. By strategically injecting artificial mistakes into clean text to mimic the specific error patterns of non-native speakers, the researchers built models that significantly outperform those trained on pristine, error-free data. Future work involves applying these training paradigms to other common error types, such as prepositions and verb agreement.
Summary updated