NAIST at 2013 CoNLL Grammatical Error Correction Shared Task

Ippei Yoshimoto, Tomoya Kose, Kensuke Mitsuzawa, Keisuke Sakaguchi, Tomoya Mizumoto, Yuta Hayashibe, Mamoru Komachi, Yuji MatsumotoCoNLL Shared Task 2013 · OpenReview

Abstract

In the CoNLL 2013 Grammatical Error Correction (GEC) shared task, the NAIST team deployed a hybrid architecture designed to address the distinct linguistic characteristics of different error types. Their system combined a syntactic Treelet Language Model for verb-related errors, a discriminative binary classifier for noun number, and Statistical Machine Translation (SMT) for prepositions and determiners. The approach emphasizes that specialized linguistic models—particularly those leveraging syntactic trees—are superior for identifying complex, long-distance grammatical dependencies compared to generic "one-size-fits-all" models.

Introduction

Grammatical Error Correction (GEC) is the automated process of detecting and fixing errors in text, primarily to assist second-language learners. The CoNLL 2013 Shared Task focused on five specific error categories: Subject-Verb Agreement (SVA), Verb Form (Vform), Noun Number (Nn), Prepositions (Prep), and Determiners (ArtOrDet).

The core challenge in GEC is that these errors operate on different linguistic levels. For instance, Subject-Verb Agreement often requires understanding long-distance dependencies where the subject and verb are separated by several words. Conversely, prepositions and determiners often depend on local, idiomatic "conventional expressions." The NAIST team addressed this by developing a three-pronged system tailored to these specific needs.

Methodology

Treelet Language Model (TLM) for Verbs

Standard n-gram models struggle with Subject-Verb Agreement when a subject and verb are separated by intervening adjectives or phrases. To solve this, the NAIST team utilized a Treelet Language Model (TLM). Instead of looking only at adjacent words, the TLM assigns probabilities to production rules within a constituent parse tree. By evaluating "treelets" (connected subgraphs), the system captures the relationship between a subject and its distant predicate regardless of the distance between them in the surface text.

Standard constituent tree vs. head-driven tree for the Treelet Language Model.

Noun Number Correction with AROW

For noun number errors (e.g., correcting "many student" to "many students"), the team employed the Adaptive Regularization of Weight Vectors (AROW) algorithm. A critical component of this classifier was the "treepath feature." This feature encodes the direction of tree traversal between the target noun and other nodes within the Noun Phrase (NP). This allows the model to understand if a noun is modified by plural-inducing words like "many" or singular-inducing words like "each."

Features used for the noun number classifier, including syntactic path features.

SMT for Prepositions and Determiners

The team treated preposition and determiner correction as a translation task, translating "learner English" into "standard English." Using the Moses decoder, they leveraged large bilingual corpora to capture idiomatic usage. To prevent the model from over-generalizing and introducing new errors, they limited the SMT output to a specific set of 48 prepositions and 25 determiners.

Datasets

The researchers utilized several key corpora for training and evaluation:

  • NUCLE (NUS Corpus of Learner English): The primary dataset for the shared task, containing essays with manual corrections.
  • Lang-8: A large-scale language-exchange corpus providing roughly 1.2 million sentence pairs for the SMT model.
  • Penn Treebank & English Gigaword: Used as "native" reference corpora to establish a baseline for correct English usage.

Key Results and Insights

The system performed competitively, ranking 4th overall in the official task run. While the original run achieved an F-score of 21.85, post-task adjustments improved this to 28.14.

Key findings included:

  • Syntax Trumps Proximity: The Treelet Language Model proved highly effective for SVA. Qualitative analysis showed the system successfully corrected errors like "This is an age which most people is retired" to "are retired," bridging the gap between "people" and "is."
  • Precision in Noun Number: The noun number classifier was particularly precise, achieving a precision of 0.4452, significantly higher than the verb or preposition subsystems.

Limitations and Future Work

The authors noted that the current pipeline treats subsystems independently. Consequently, an error in noun number could negatively impact the correction of subject-verb agreement. Future research will focus on "joint correction" models to handle these interdependencies and improving Part-of-Speech (POS) tagging, which currently acts as a bottleneck for the syntactic models.

Reference: Yoshimoto, I., Kose, T., Mitsuzawa, K., Sakaguchi, K., Mizumoto, T., Hayashibe, Y., Komachi, M., & Matsumoto, Y. (2013). NAIST at 2013 CoNLL Grammatical Error Correction Shared Task. Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, 26–33.

Summary updated

BibTeX
@inproceedings{yoshimoto2013naist,
  author = {Ippei Yoshimoto and Tomoya Kose and Kensuke Mitsuzawa and Keisuke Sakaguchi and Tomoya Mizumoto and Yuta Hayashibe and Mamoru Komachi and Yuji Matsumoto},
  title = {{NAIST at 2013 CoNLL Grammatical Error Correction Shared Task}},
  booktitle = {CoNLL Shared Task},
  year = {2013},
  url = {https://openreview.net/forum?id=HPkU-xXwXW_},
}
Figure