Language Model Based Grammatical Error Correction without Annotated Training Data

Christopher Bryant, Ted BriscoeNAACL 2018

TL;DR

This paper revisits language model (LM) based approaches for Grammatical Error Correction (GEC), demonstrating that a simple LM system can achieve competitive results with minimal annotated data. This approach is particularly valuable for low-resource languages where extensive labeled data is unavailable.

What Problem Does This Paper Address?

The field of Grammatical Error Correction (GEC) has largely moved towards complex Statistical Machine Translation (SMT) and neural network-based systems, which typically require vast amounts of annotated training data. While these methods have achieved state-of-the-art results, they are inaccessible for languages with limited parallel corpora. This paper addresses the gap by re-evaluating simpler Language Model (LM) based GEC approaches, aiming to show their viability and competitiveness even with minimal annotated data, making GEC more accessible for a wider range of languages.

Methods

The core of the proposed method is based on the idea that low-probability sequences are more likely to contain grammatical errors. The system iteratively corrects errors by:

  1. Calculating normalized log probabilities: The system first assesses the probability of the input sentence.
  2. Building confusion sets: For each token in the sentence, potential replacement candidates are identified. These candidates are generated using:
  3. CyHunspell: For non-word errors (misspellings, inflectional errors).
  4. Automatically Generated Inflection Database (AGID): For morphological errors (noun number, verb tense, adjective form).
  5. Manually defined sets: For articles and prepositions, including a null character for deletions.
  6. Re-scoring with candidates: The system re-scores the sentence by substituting each token with its candidates.
  7. Applying corrections: A correction is applied if it significantly increases the sentence's probability above a defined threshold.
  8. Iteration: Steps 1-4 are repeated to address errors that might interact and influence each other.

A key aspect is that the confusion sets for most error types are generated without relying on annotated training data, except for manually defined sets for articles and prepositions.

Datasets

The paper uses several standard GEC corpora for evaluation:

  • CoNLL-2013 and CoNLL-2014: Widely used benchmarks for GEC.
  • FCE (First Certificate in English): A public dataset.
  • JFLEG (Fluency Corpus): A dataset focused on fluency.

The language model used for experiments was trained on the One Billion Word Benchmark dataset using KenLM.

Key Insights and Ideas

  • LM-based GEC is still competitive: Despite the trend towards more complex models, a simple LM-based approach can achieve results comparable to state-of-the-art systems, especially when annotated data is scarce.
  • Minimal annotation is sufficient for tuning: The system requires only a small amount of annotated data (around 1000 sentences) for tuning the probability threshold, making it practical for low-resource scenarios.
  • Error interaction necessitates iteration: The iterative correction process is crucial because correcting one error can sometimes enable or improve the correction of another, as demonstrated by the example in Table 1.
  • Dataset characteristics matter: The optimal probability threshold varies significantly across datasets, likely due to differences in error type distributions (e.g., spelling errors vs. fluency errors).
  • Fluency-focused datasets benefit LM approaches: The system performed surprisingly well on the JFLEG dataset, which was corrected for fluency, suggesting that LM-based methods might be particularly effective when the reference corrections prioritize fluency.

Key Results

  • The proposed LM-based system achieved competitive results on several benchmark datasets, outperforming previous LM-based approaches and even some state-of-the-art systems on JFLEG-test.
  • On CoNLL-2014, the system placed fourth overall, significantly narrowing the gap between third and fourth place compared to previous LM-based systems.
  • The system's performance on JFLEG-test was particularly strong, coming within 2 F0.5 of the top system, despite only correcting a limited subset of error types.
  • A post-processing step of capitalizing the first character of sentences led to notable improvements, especially on JFLEG-test (over 5 F0.5).

Limitations and Future Work

  • Limited error type coverage: The current system only corrects a specific set of error types (non-words, morphology, articles, prepositions) and does not handle missing words, which constitute a significant portion of errors.
  • Neural language models: The authors suggest exploring neural language modeling techniques for potentially better performance.
  • Threshold optimization: Future work could involve setting error-type-specific thresholds rather than a global one.
  • Handling missing words: Incorporating a mechanism to correct missing words is a key area for future development.

Summary updated

BibTeX
@inproceedings{bryant2018language,
  author = {Christopher Bryant and Ted Briscoe},
  title = {{Language Model Based Grammatical Error Correction without Annotated Training Data}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure