Correction Detection and Error Type Selection as an ESL Educational Aid

Ben Swanson, Elif YamangilNAACL 2012

TL;DR

This paper introduces a novel NLP task: automatically identifying and classifying the types of errors teachers correct in student essays. They developed a log-linear classifier trained on the Cambridge Learner Corpus, achieving high accuracy in distinguishing between various error categories, which can serve as a valuable educational aid for both students and teachers.

What Problem Does This Paper Address?

In foreign language education, teachers spend significant time identifying and correcting errors in student essays, ranging from grammar and spelling to style and semantics. Providing targeted feedback is crucial for student learning, but manually isolating and categorizing each correction is time-consuming. This paper addresses the challenge of automating this process, aiming to create a tool that can assist teachers by automatically detecting individual corrections and classifying them into linguistically meaningful error types. This would allow for more efficient analysis of student weaknesses and inform lesson planning.

Methods

The authors propose a two-component NLP system. First, they tackle the problem of correction detection, which involves isolating individual edits made by a teacher between an original and a corrected sentence. They employ a heuristic strategy based on edit distance operations (Insert, Delete, Substitute, Equal) to segment corrections. Second, they focus on error type selection, where each detected correction is classified into one of 75 predefined linguistic error categories. For this classification, they utilize a log-linear model trained with a rich set of features. These features include:

  • Edit distance operations: Indicators for Insert, Delete, Substitute, and their associated words.
  • POS Context: Part-of-speech tags surrounding the edit in the corrected sentence.
  • Targeted substitution features: Information about whether words are in a dictionary (WordNet), share the same stem, or have similar suffixes.
  • Global features: Such as the total number of edits or whether one sentence is a word-reordering of another.

They also explore a compressed set of 15 error types for simpler classification.

Datasets

The primary dataset used is the Cambridge Learner Corpus (CLC). This corpus contains approximately 1200 essays with corrections annotated in XML, including specific error type tags. The authors extracted sentence pairs (original and corrected) and created instances of (original sentence, corrected sentence, error type) triples, resulting in a dataset of 45,080 instances for training and evaluation.

Key Insights and Ideas

  • Automating teacher feedback: The core idea is to leverage NLP to reduce the manual burden on teachers, providing them with actionable insights into student errors.
  • Correction detection as a distinct task: The paper highlights that isolating individual corrections is a crucial prerequisite for accurate error type classification, and proposes heuristic methods for this.
  • Linguistically motivated features: The success of their classifier hinges on a carefully designed feature set that captures various aspects of the edit, from word-level information to syntactic context.
  • Compressed vs. full error types: The study shows that a compressed set of 15 error types can achieve high accuracy, suggesting a practical balance between granularity and classification performance.

Key Results

  • The log-linear classifier achieved 88.9% accuracy for the 15-class problem and 83.8% accuracy for the full 75-class problem.
  • These accuracies significantly outperform baseline methods, which achieved around 33.3% and 7.8% respectively.
  • The authors found that their heuristic h0, which merges consecutive non-Equal edits, provided the best F-scores for predicting both correction boundaries and labels.
  • Removing strongly lexicalized features led to a reduction in accuracy from 88.9% to 82.4% for the 15-class problem, indicating the importance of lexical information.

Limitations and Future Work

  • Long-range lexical dependencies: The authors acknowledge that their system struggles with capturing long-range lexical dependencies, which can affect the accuracy of agreement error detection.
  • Heuristic correction detection: While effective, the heuristic correction detection strategy could be replaced with a more robust algorithm.
  • Future directions:
    • More robust correction detection: Developing more sophisticated algorithms for identifying correction boundaries.
  • Incorporating syntactic information: Using synchronous grammars to better handle agreement errors.
  • Broader applications: Exploring the use of this system to aid automated error correction systems.
  • Real-world evaluation: Testing the application in actual classroom settings to determine an optimal set of error types that are most useful for teachers.

Summary updated

BibTeX
@inproceedings{swanson2012correction,
  author = {Ben Swanson and Elif Yamangil},
  title = {{Correction Detection and Error Type Selection as an ESL Educational Aid}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2012},
}
Figure