Automated Whole Sentence Grammar Correction Using a Noisy Channel Model

Y. Albert Park, Roger LevyACL 2011 · OpenReview

This paper, presented at ACL 2011, offers a classic yet innovative take on Natural Language Processing (NLP) by applying the "noisy channel" framework—originally popularized in speech recognition and machine translation—to the task of automated grammar correction.

Abstract

Automated grammar correction is often approached through rule-based systems or specific error classifiers. Park and Levy propose a whole-sentence correction system using a noisy channel model implemented via Weighted Finite-State Transducers (wFSTs). By utilizing the Expectation-Maximization (EM) algorithm, they successfully train the model on a large corpus of erroneous ESL essays without requiring expensive, manually corrected "parallel" data. Their system outperforms standard tools like Aspell in contextual spelling and grammar tasks.

The Problem: Narrow-Minded Correction

Most grammar correction tools at the time were limited in scope. They were either rule-based (searching for specific violations like "subject-verb disagreement") or classifier-based (focusing on a single error type, like prepositions). These approaches often ignore the wider sentence context and fail when a sentence contains multiple, overlapping errors.

Furthermore, training high-quality correction models usually requires "parallel corpora"—pairs of incorrect sentences and their human-corrected versions. These are notoriously difficult and expensive to produce. The authors sought a way to build a robust, context-aware system that could learn how humans make mistakes using only the "noisy" (erroneous) data itself.

The Noisy Channel Framework

The authors frame grammar correction as a Noisy Channel Model. The core intuition is that a writer intends to produce a "clean" sentence ($S_{orig}$), but it passes through a "noisy channel" (the writer's cognitive or typing errors) and emerges as an observed, erroneous sentence ($S_{obs}$).

The Noisy Channel Model framework for grammar correction.

To fix the error, the system finds the $S_{orig}$ that maximizes the probability $P(S_{obs}|S_{orig})P(S_{orig})$. This involves two main components:

  1. Base Language Model: A bigram model trained on the British National Corpus representing "correct" English.
  2. Noise Model: A series of layered modules that simulate different human errors, including spelling (Damerau-Levenshtein distance), articles (confusion between a, an, and the), prepositions, and wordforms (tense/number inflections).

Technical Implementation

The entire system is built using Weighted Finite-State Transducers (wFSTs), allowing for efficient composition of the various noise modules and the language model.

A key innovation is the use of Unsupervised Learning. The authors use the Expectation-Maximization (EM) algorithm to learn the parameters of the noise model. By analyzing thousands of errors in a corpus of ~478,000 sentences from Korean ESL essays, the model "guesses" the most likely intended words, iteratively refining its understanding of common mistake patterns without needing human-corrected references for training.

Key Results

The model was evaluated against human corrections obtained via Amazon Mechanical Turk, using Machine Translation metrics like BLEU and METEOR.

Performance comparison between Aspell and the proposed spelling noise model.
  • Contextual Spelling: The spelling-only noise model outperformed the GNU Aspell checker. Because the model considers the whole sentence, it can resolve ambiguities where a misspelling results in another valid word (e.g., correcting "minuets" to "minutes" in the context of time).
  • Incremental Gains: Adding article and wordform modules further improved performance. The wordform model saw the highest jump in sentences with improved scores.

Limitations and Future Work

Despite its successes, the model's reliance on a bigram language model is a significant limitation. Because it only considers two words at a time, it struggles with long-distance dependencies and can be biased toward shorter sentences. Additionally, the preposition model occasionally decreased performance by being too aggressive without sufficient context. Future directions include moving toward more complex language models (like trigrams or PCFGs) and adding models for word deletion and reordering.

Summary updated

BibTeX
@inproceedings{park2011automated,
  author = {Y. Albert Park and Roger Levy},
  title = {{Automated Whole Sentence Grammar Correction Using a Noisy Channel Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2011},
  url = {https://openreview.net/forum?id=Hk-EdjldbB},
}
Figure