Improving Precision of Grammatical Error Correction with a Cheat Sheet

Mengyang Qiu, Xuejiao Chen, Maggie Liu, Krishna Parvathala, Apurva Patil, Jungyeul ParkACL 2019

TL;DR

This paper introduces a "cheat sheet" approach to improve grammatical error correction (GEC) by augmenting training data with error-focused phrases. By directly exposing models to error-context pairs, they achieve higher precision, particularly for syntactic errors, outperforming baseline models and demonstrating the limitations of traditional SMT for GEC.

What Problem Does This Paper Address?

Grammatical Error Correction (GEC) is a crucial NLP task for helping language learners and improving text quality. While neural machine translation (NMT) and sequence-to-sequence models have shown promise, effectively training these models is challenging due to the limited quantity and complexity of errors in typical training data. Specifically, sentences with multiple errors can be difficult for models to learn and generalize from. This paper addresses the challenge of improving the precision of GEC systems, aiming to reduce false positives (incorrectly flagged or corrected errors) by providing models with more targeted learning signals.

Methods

The core contribution is the "cheat sheet" method, which augments existing GEC training data with error-focused phrases. Two primary methods for generating these phrases are explored:

  1. Error+Context Dictionary: This method extracts phrases directly from GEC corpora (specifically, the "m2 files" which contain error annotations). Each extracted phrase includes the erroneous word and one context word on either side. This approach preserves the natural distribution of errors found in real learner data.
  2. SMT Phrase Translation Table: This method leverages phrases extracted from statistical machine translation (SMT) phrase tables. These phrases are typically longer sequences of words and can contain errors in various positions, not just centered.

For the unrestricted GEC track, they also generate a "pseudo corpus" by injecting specific types of syntactic errors (e.g., verb form, noun inflection) into clean sentences, using the error+context dictionary as a source for these errors.

The authors then train both SMT (using Moses) and NMT (a 7-layer convolutional seq2seq model) on these augmented datasets.

Datasets

  • BEA 2019 Shared Task Datasets: The paper uses four datasets provided for the BEA 2019 Shared Task on GEC: FCE, NUCLE, W&I+LOCNESS, and Lang-8. This forms the baseline training data.
  • AFP News Corpus: A 3-gram language model was trained on 2 million sentences from this corpus for the SMT setup.
  • Pseudo Corpus: A synthetic dataset generated by injecting syntactic errors into clean sentences. This corpus, combined with the original data, forms the training set for the unrestricted track.

Key Insights and Ideas

  • Direct Error Exposure is Key: Simply providing error-context phrases directly from GEC corpora significantly outperforms using phrases from SMT tables. This suggests that the "naturalness" and directness of error representation are crucial.
  • Syntactic Errors are More Tractable: The "cheat sheet" and pseudo corpus approach showed more significant improvements for syntactic errors compared to semantic errors, which require deeper meaning understanding.
  • NMT Benefits More from Augmentation: While SMT saw limited gains, NMT models demonstrated a clear benefit from the augmented data, especially in the unrestricted track. This highlights NMT's greater capacity to leverage richer training signals.
  • Precision Over Recall: The proposed methods notably improved precision, but recall remained lower. This indicates the models became more conservative, correctly identifying fewer errors but also making fewer mistakes.

Key Results

  • Error+Context Outperforms SMT Phrases: Phrases directly extracted from GEC corpora ("error+context") yielded significantly better results than those from SMT phrase tables.
  • Precision Improvement: The "cheat sheet" approach led to higher precision and F0.5 scores in both restricted and unrestricted tracks compared to baseline models.
  • NMT Dominance: GEC-tuned NMT models consistently outperformed generic SMT models.
  • Unrestricted Track Gains: The additional pseudo corpus for the unrestricted track further boosted performance, particularly for NMT, showing gains in precision, recall, and F0.5. For example, the F0.5 gap in the unrestricted setting for NMT was over 5%.
  • Specific Error Type Improvement: For the VERB:FORM error type, NMT precision and F0.5 almost doubled when trained with the augmented data compared to SMT.

Limitations and Future Work

  • Low Recall: A significant limitation is the consistently low recall across all settings. The authors attribute this to models being too conservative, possibly due to a large proportion of unchanged sentences in the training data.
  • SMT Phrase Table Issues: The SMT phrase table approach was detrimental, likely due to partial redundancies in the phrases. Future work could involve investigating pruning techniques for SMT phrase tables.
  • Focus on Syntactic Errors: The study primarily focused on syntactic errors. Future work should explore semantic errors, which are more complex and require deeper contextual understanding.
  • Incorporating ELMo: The authors suggest investigating the incorporation of ELMo (Embeddings from Language Models) into their NMT model to improve the handling of semantic errors, as ELMo can capture contextual word meaning variations.
  • Recall vs. Proportion of Unchanged Sentences: Further exploration of the relationship between recall and the proportion of unchanged sentences in GEC is planned.

Summary updated

BibTeX
@inproceedings{qiu2019improving,
  author = {Mengyang Qiu and Xuejiao Chen and Maggie Liu and Krishna Parvathala and Apurva Patil and Jungyeul Park},
  title = {{Improving Precision of Grammatical Error Correction with a Cheat Sheet}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure