Neural Sequence-Labelling Models for Grammatical Error Correction

Helen Yannakoudakis, Marek Rei, Øistein E. Andersen, Zheng YuanEMNLP 2017 · OpenReview

Abstract

This paper presents a novel approach to Grammatical Error Correction (GEC) by leveraging neural sequence-labelling models to re-rank the outputs of traditional Statistical Machine Translation (SMT) systems. By employing a bidirectional LSTM to detect errors at the token level and incorporating character-level gating, the proposed method achieves state-of-the-art results on multiple benchmarks using a minimal set of non-linguistic features.

Introduction

Grammatical Error Correction (GEC) is frequently framed as a translation task: converting ungrammatical "learner" English into correct "native" English. While Statistical Machine Translation (SMT) has historically been the standard approach, it often fails to capture the subtle nuances of grammaticality. Previous attempts to enhance SMT systems involved re-ranking the top $N$ hypotheses using complex, hand-crafted linguistic features such as part-of-speech tags or dependency parses. This work introduces a more streamlined, data-driven approach that utilizes neural networks to score hypotheses based on their likelihood of containing errors.

Neural Sequence-Labelling Architecture

The core of this methodology is a Neural Sequence-Labelling Model designed specifically for error detection rather than direct translation.

Neural sequence-labelling architecture featuring character-level Bi-LSTM gating and multi-task learning components.
  1. Bi-LSTM Network: The model uses a Bidirectional Long Short-Term Memory (Bi-LSTM) network to predict the probability of each token being "correct" or "incorrect" within its context.
  2. Character-Level Gating: To address out-of-vocabulary words and misspellings, a character-level Bi-LSTM is integrated. An adaptive gating mechanism combines these character-level insights with standard word embeddings, allowing the model to analyze the internal structure of words.
  3. Multi-task Learning: The model is simultaneously trained as a Language Model, predicting adjacent words. This auxiliary task improves the internal representation of English grammar.

Re-ranking Strategy

The neural model scores the top 10 hypotheses from an SMT system using three primary features:

  • Sentence Probability: The sum of log-probabilities for all tokens being "correct."
  • Levenshtein Distance (LD): A measure of the alignment between SMT corrections and the neural model's predicted error map.
  • True/False Positives: A ratio comparing the SMT's changes against the neural model's flagged errors.

Experimental Results

The hybrid SMT + Neural Re-ranker was evaluated on several standard GEC benchmarks, including the FCE (exam essays), NUCLE (advanced student essays), and JFLEG (fluency-focused edits).

Performance comparison of the hybrid SMT + Neural Re-ranker against various baselines across the FCE, CoNLL, and JFLEG datasets.

The system achieved state-of-the-art performance across all test sets. On the FCE dataset, the $F_{0.5}$ score improved from a baseline of 52.90 to 55.60. Significant gains were also observed in the CoNLL 2014 benchmark (37.33 to 42.44) and the JFLEG GLEU score (46.10 to 47.72). Ablation studies confirmed that Sentence Probability was the most influential feature in the re-ranking process.

Discussion and Conclusion

One of the most significant insights of this research is that a model trained purely for error detection can be highly effective at selecting optimal corrections. By identifying which SMT hypothesis contains the fewest remaining predicted errors, the system significantly outperforms raw SMT output. Furthermore, the model's simplicity—relying on only four features—makes it more portable and efficient than previous feature-heavy re-rankers.

However, the system is constrained by the "N-best ceiling"; if the correct correction is not within the SMT's top 10 hypotheses, the re-ranker cannot select it. Future work may involve integrating the neural detection model directly into the SMT decoding process to overcome this limitation and further refine performance on specific error types like contractions and verb forms.

Summary updated

BibTeX
@inproceedings{yannakoudakis2017neural,
  author = {Helen Yannakoudakis and Marek Rei and Øistein E. Andersen and Zheng Yuan},
  title = {{Neural Sequence-Labelling Models for Grammatical Error Correction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=HyVHrGGubr},
}
Figure