Algorithm Selection and Model Adaptation for ESL Correction Tasks

Alla Rozovskaya, Dan RothACL 2011 · OpenReview

Abstract

We consider the problem of correcting errors made by English as a Second Language (ESL) writers and address two issues that are essential to making progress in ESL error correction: algorithm selection and model adaptation to the first language of the ESL learner. A variety of learning algorithms have been applied to correct ESL mistakes, but often comparisons were made between incomparable data sets. We conduct an extensive, fair comparison of four popular learning methods for the task, reversing conclusions from earlier evaluations. Furthermore, we propose a novel way to adapt a learned algorithm to the first language of the writer that is both cheaper to implement and performs better than other adaptation methods.

Introduction

Writing errors involving prepositions and articles are systematic among ESL learners but remain difficult to automate. Two major hurdles persist in the field: inconsistent evaluation methodologies and the influence of a writer's first language (L1), known as L1 interference. Previous studies often compared algorithms using vastly different training sets and features, obscuring which methods actually perform best. Additionally, most models are trained on native English, ignoring the predictable error patterns specific to a learner's native background (e.g., Chinese vs. Russian speakers).

Methodology

The "Fair" Comparison

To establish a baseline, we evaluated four linear learning models on identical datasets and feature sets:

  • Averaged Perceptron (AP): A discriminative, mistake-driven algorithm.
  • Naive Bayes (NB): A generative probabilistic model.
  • Language Model (LM): Predicts word probability based on context.
  • SumLM: A count-based method summing logs of n-gram counts.

Model Adaptation (NB-adapted)

We introduce a method to inject L1 knowledge by modifying the prior probabilities of a Naive Bayes classifier. These adapted priors are calculated from a small sample of annotated ESL data, conditioned on the writer's native language and the specific preposition used. This approach avoids the complexity of generating artificial errors or retraining models from scratch.

Experimental Results

Algorithm Performance and Data Scale

Our results indicate that previous claims favoring Language Models or count-based methods were largely artifacts of those models being trained on significantly more data. When evaluated on equal footing, the Averaged Perceptron (AP) consistently outperforms other methods. Notably, AP trained on 5 million contexts performs as well as Naive Bayes trained on 10 million, and it outperforms SumLM even when the latter has a 10x data advantage.

Precision-Recall curves comparing algorithms across different training set sizes, demonstrating that AP maintains superior performance even with less data.

Effectiveness of L1 Adaptation

The proposed NB-adapted model significantly outperformed the standard AP and even an adapted version of AP that utilized artificial error generation. By adjusting priors to reflect the likelihood of specific L1-influenced mistakes, the model achieved substantial gains in precision with minimal loss in recall. For instance, on the WikiNYT dataset, adaptation improved the Average Area Under the Curve (AAUC) for Naive Bayes from 29 to 53.

Precision-Recall curves for adapted models, showing that the NB-adapted approach significantly outperforms standard models and other adaptation techniques.

Conclusion

This study demonstrates that discriminative models like the Averaged Perceptron are superior for ESL correction when data is controlled. Furthermore, we show that L1 adaptation does not require massive synthetic datasets; a simple shift in Naive Bayes priors using a small amount of annotated ESL data is both highly efficient and effective. Future work may explore the application of this prior-shifting method to more complex syntactic issues such as subject-verb agreement.

Summary updated

Abstract

We consider the problem of correcting errors made by English as a Second Language (ESL) writers and address two issues that are essential to making progress in ESL error correction - algorithm selection and model adaptation to the first language of the ESL learner. A variety of learning algorithms have been applied to correct ESL mistakes, but often comparisons were made between incomparable data sets. We conduct an extensive, fair comparison of four popular learning methods for the task, reversing conclusions from earlier evaluations. Our results hold for different training sets, genres, and feature sets. A second key issue in ESL error correction is the adaptation of a model to the first language of the writer. Errors made by non-native speakers exhibit certain regularities and, as we show, models perform much better when they use knowledge about error patterns of the non-native writers. We propose a novel way to adapt a learned algorithm to the first language of the writer that is both cheaper to implement and performs better than other adaptation methods.
BibTeX
@inproceedings{rozovskaya2011algorithm,
  author = {Alla Rozovskaya and Dan Roth},
  title = {{Algorithm Selection and Model Adaptation for ESL Correction Tasks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2011},
  url = {https://openreview.net/forum?id=r1NCssgdWS},
}
Figure