Abstract
In the CoNLL 2013 Grammatical Error Correction (GEC) shared task, the NAIST team deployed a hybrid architecture designed to address the distinct linguistic characteristics of different error types. Their system combined a syntactic Treelet Language Model for verb-related errors, a discriminative binary classifier for noun number, and Statistical Machine Translation (SMT) for prepositions and determiners. The approach emphasizes that specialized linguistic models—particularly those leveraging syntactic trees—are superior for identifying complex, long-distance grammatical dependencies compared to generic "one-size-fits-all" models.
Introduction
Grammatical Error Correction (GEC) is the automated process of detecting and fixing errors in text, primarily to assist second-language learners. The CoNLL 2013 Shared Task focused on five specific error categories: Subject-Verb Agreement (SVA), Verb Form (Vform), Noun Number (Nn), Prepositions (Prep), and Determiners (ArtOrDet).
The core challenge in GEC is that these errors operate on different linguistic levels. For instance, Subject-Verb Agreement often requires understanding long-distance dependencies where the subject and verb are separated by several words. Conversely, prepositions and determiners often depend on local, idiomatic "conventional expressions." The NAIST team addressed this by developing a three-pronged system tailored to these specific needs.
Methodology
Treelet Language Model (TLM) for Verbs
Standard n-gram models struggle with Subject-Verb Agreement when a subject and verb are separated by intervening adjectives or phrases. To solve this, the NAIST team utilized a Treelet Language Model (TLM). Instead of looking only at adjacent words, the TLM assigns probabilities to production rules within a constituent parse tree. By evaluating "treelets" (connected subgraphs), the system captures the relationship between a subject and its distant predicate regardless of the distance between them in the surface text.
Noun Number Correction with AROW
For noun number errors (e.g., correcting "many student" to "many students"), the team employed the Adaptive Regularization of Weight Vectors (AROW) algorithm. A critical component of this classifier was the "treepath feature." This feature encodes the direction of tree traversal between the target noun and other nodes within the Noun Phrase (NP). This allows the model to understand if a noun is modified by plural-inducing words like "many" or singular-inducing words like "each."
SMT for Prepositions and Determiners
The team treated preposition and determiner correction as a translation task, translating "learner English" into "standard English." Using the Moses decoder, they leveraged large bilingual corpora to capture idiomatic usage. To prevent the model from over-generalizing and introducing new errors, they limited the SMT output to a specific set of 48 prepositions and 25 determiners.
Datasets
The researchers utilized several key corpora for training and evaluation:
- NUCLE (NUS Corpus of Learner English): The primary dataset for the shared task, containing essays with manual corrections.
- Lang-8: A large-scale language-exchange corpus providing roughly 1.2 million sentence pairs for the SMT model.
- Penn Treebank & English Gigaword: Used as "native" reference corpora to establish a baseline for correct English usage.
Key Results and Insights
The system performed competitively, ranking 4th overall in the official task run. While the original run achieved an F-score of 21.85, post-task adjustments improved this to 28.14.
Key findings included:
- Syntax Trumps Proximity: The Treelet Language Model proved highly effective for SVA. Qualitative analysis showed the system successfully corrected errors like "This is an age which most people is retired" to "are retired," bridging the gap between "people" and "is."
- Precision in Noun Number: The noun number classifier was particularly precise, achieving a precision of 0.4452, significantly higher than the verb or preposition subsystems.
Limitations and Future Work
The authors noted that the current pipeline treats subsystems independently. Consequently, an error in noun number could negatively impact the correction of subject-verb agreement. Future research will focus on "joint correction" models to handle these interdependencies and improving Part-of-Speech (POS) tagging, which currently acts as a bottleneck for the syntactic models.
Reference: Yoshimoto, I., Kose, T., Mitsuzawa, K., Sakaguchi, K., Mizumoto, T., Hayashibe, Y., Komachi, M., & Matsumoto, Y. (2013). NAIST at 2013 CoNLL Grammatical Error Correction Shared Task. Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, 26–33.
Summary updated