Abstract
This paper describes the system of Shanghai Jiao Tong University team in the CoNLL-2014 shared task. Error correction operations are encoded as a group of predefined labels and therefore the task is formulized as a multi-label classification task. For training, labels are obtained through a strict rule-based approach. For decoding, errors are detected and corrected according to the classification results. A single maximum entropy model is used for the classification implementation incorporated with an improved feature selection algorithm. Our system achieved precision of 29.83, recall of 5.16 and F 0.5 of 15.24 in the official evaluation.
Introduction
Grammatical Error Correction (GEC) is a complex task due to the wide variety of error types, ranging from simple orthographic mistakes to nuanced syntactic issues like subject-verb agreement or article usage. Historically, state-of-the-art systems relied on ensembles of specialized sub-models, each tailored to a specific error category. While effective, this modular approach introduces significant maintenance overhead and potential conflicts between sub-models. This paper presents a unified approach developed for the CoNLL-2014 shared task, utilizing a single Maximum Entropy (ME) classifier to handle 28 different error types simultaneously.
Methodology
The core innovation lies in reframing GEC as a multi-label classification problem. The process begins with an automated labeling system based on an extended Levenshtein edit distance. Unlike standard edit distance, this version accounts for linguistic inflections (e.g., changing "go" to "went"). By comparing learner sentences with corrected versions, the system extracts specific edit operations.
These operations are then mapped to a predefined set of labels. For instance, changing a noun to its plural form results in the NPLURAL label, while adding a definite article results in THE+. Most words, requiring no change, are assigned a null symbol ($\odot$).
A single Maximum Entropy model is employed to predict these labels. ME models are particularly well-suited for NLP tasks as they can integrate a vast number of overlapping features without requiring naive independence assumptions. To optimize the model, the team implemented a "recruit and shakeoff" greedy feature selection algorithm. Starting with 109 candidate features—including POS tags and dependency relations—the algorithm iteratively refined the set to 220 combined features, maximizing performance while minimizing redundancy.
Datasets and Training
The system was trained on the CoNLL-2014 shared task dataset, derived from the NUS Corpus of Learner English (NUCLE). This dataset contains approximately 1,400 articles written by non-native speakers. To enhance training efficiency and model focus, the authors filtered the training data to include only sentences containing errors. This reduced the training set to 38% of its original size but led to improved performance on the blind evaluation set of 50 articles.
Key Results and Insights
The integrated model approach successfully avoids the "error propagation" common in multi-model pipelines. By predicting the correction directly as a label, the system collapses error detection and correction into a single step. The authors observed that feature synergy was crucial; complex combinations of atomic features (e.g., the current word paired with the POS tag of a preceding word) provided the most significant gains.
In the official CoNLL-2014 evaluation, the system achieved a precision of 29.83%, a recall of 5.16%, and an F0.5 score of 15.24%. The high precision relative to recall reflects a conservative correction strategy, prioritizing accuracy over coverage—a desirable trait for educational applications.
As shown in the performance metrics, the "sys_220" model, which utilized the full set of 220 optimized features, significantly outperformed versions with fewer features (sys_10 and sys_55), validating the effectiveness of the feature selection process.
Conclusion and Future Work
While the single-model approach demonstrates efficiency and high precision, the low recall remains a primary challenge. This is largely attributed to label sparsity; of the 7,000 unique edit labels generated, many occurred too infrequently to be modeled effectively. The final system was pruned to 109 common operations, leaving a "long tail" of rare errors unaddressed. Future research will focus on improving the handling of these sparse labels and addressing inconsistencies in human annotations to provide a more robust training signal for the classifier.
Summary updated