The CoNLL-2013 Shared Task on Grammatical Error Correction

Hwee Tou Ng, Siew Mei Wu, Yuanbin Wu, Christian Hadiwinoto, Joel R. TetreaultarXiv 2025 · DOI · arXiv

Abstract

The CoNLL-2013 shared task represents a pivotal shift in Grammatical Error Correction (GEC) from isolated sub-problems to integrated, end-to-end systems. By focusing on five common error types in learner essays—Articles/Determiners, Prepositions, Noun Number, Verb Form, and Subject-Verb Agreement—the task established a robust benchmark using the NUCLE corpus and the MaxMatch ($M^2$) evaluation metric. This paper reviews the task's design, the diverse methodologies employed by seventeen international teams, and the key findings regarding the complexity of interacting errors and the subjectivity of linguistic correction.

Introduction

Grammatical Error Correction (GEC) is a critical application of Natural Language Processing, yet it remains one of the most challenging due to the contextual and subjective nature of "correctness." Previous shared tasks, such as HOO 2011 and 2012, were limited in scope, often targeting only one or two error types. The CoNLL-2013 shared task was designed to push the field toward more holistic systems capable of correcting multiple, often interacting, error types in essays written by English as a Second Language (ESL) learners.

Task Definition and Error Types

Participants were tasked with identifying and correcting five specific categories of grammatical errors. These categories were selected based on their frequency and impact on the writing of non-native speakers.

Examples of the five grammatical error types addressed in the shared task.

The five error types included:

  1. ArtOrDet: Article or determiner errors.
  2. Prep: Preposition errors.
  3. Nn: Noun number errors.
  4. Vform: Verb form errors.
  5. SVA: Subject-verb agreement errors.

Methodology

The task introduced two major components to the GEC community: the NUCLE corpus and the $M^2$ scorer.

The NUS Corpus of Learner English (NUCLE) served as the primary training set, comprising approximately 1.2 million tokens from over 1,400 essays. A blind test set of 50 essays was used for final evaluation. To address the limitations of rigid string-matching, the MaxMatch ($M^2$) scorer was implemented. Unlike previous metrics that relied on specific edit sequences, $M^2$ searches for a sequence of system edits that maximally matches the gold-standard edits, accommodating multiple valid correction paths for a single error.

System Approaches

Seventeen teams participated, utilizing a variety of linguistic features and external resources. Approaches generally fell into two paradigms: classification-based models (e.g., MaxEnt, SVM, or Naive Bayes) and Machine Translation (MT) frameworks.

Summary of participant approaches, including error types handled and linguistic features used.

Classification systems treated GEC as a supervised labeling task, often relying heavily on POS tags and dependency parses. MT-based systems treated "erroneous" English as a source language to be translated into "correct" English, an approach that naturally handles interacting errors—such as simultaneously correcting a noun's number and its corresponding verb's agreement.

Results and Discussion

The top-performing system, developed by the UIUC team, achieved an $F_1$ score of 31.20% on the original gold standard. A significant finding was the impact of "alternative" valid corrections. When human-vetted alternative answers were included in the evaluation, the top $F_1$ score rose to 42.14%, highlighting the inherent subjectivity in grammar correction.

Performance varied significantly across error types. Noun Number (Nn) errors were generally the easiest to correct, while Prepositions (Prep) proved the most difficult. Most systems prioritized precision over recall, reflecting a design philosophy that it is better to leave an error uncorrected than to introduce a "false correction."

Conclusion

The CoNLL-2013 shared task successfully transitioned GEC toward more comprehensive system evaluations. However, the gap between the best automated systems (~42% $F_1$) and human performance remains wide. Future work must address a broader range of error types, such as word choice and punctuation, and develop models capable of deeper semantic understanding to handle the nuances of human language.

Summary updated

BibTeX
@misc{ng2025conll,
  author = {Hwee Tou Ng and Siew Mei Wu and Yuanbin Wu and Christian Hadiwinoto and Joel R. Tetreault},
  title = {{The CoNLL-2013 Shared Task on Grammatical Error Correction}},
  year = {2025},
  eprint = {2507.09474},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.09474},
}
Figure