Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data

Roman Grundkiewicz, Marcin Junczys-Dowmunt, Kenneth HeafieldACL 2019

TL;DR

This paper tackles the data sparsity problem in neural grammatical error correction (GEC) by introducing an unsupervised method to generate synthetic training data using spellchecker confusion sets. This synthetic data is then used to pre-train Transformer models, leading to state-of-the-art results on major GEC benchmarks, particularly in low-resource scenarios.

What Problem Does This Paper Address?

Neural Grammatical Error Correction (GEC) systems, which aim to automatically correct errors in written text, have seen significant progress by adopting techniques from neural machine translation (NMT). However, a major bottleneck is the scarcity of high-quality, large-scale parallel data (sentences with errors and their corrected versions). Existing methods to address this data sparsity include generating synthetic errors, using monolingual data for pre-training, or leveraging large out-of-domain parallel corpora. This paper focuses on improving GEC systems, especially in scenarios where limited error-annotated data is available, by proposing a novel and effective unsupervised method for generating synthetic training data.

Methods

The core of this work lies in two main areas:

  • Unsupervised Synthetic Data Generation: Instead of relying on complex probabilistic models or existing error-annotated corpora to generate synthetic errors, the authors propose a simpler approach. They leverage confusion sets from a standard spellchecker (Aspell) to identify words that are commonly confused. For each sentence in an error-free corpus, they randomly select words and replace them with a word from their corresponding confusion set. This process is controlled by a sampling probability to mimic realistic error rates. Additionally, they introduce noise into source words by randomly perturbing characters to make the models more robust to spelling errors. This method is unsupervised, meaning it doesn't require any pre-labeled error data.
  • Model Pre-training and Fine-tuning: The generated synthetic parallel data is used to pre-train Transformer sequence-to-sequence models. The authors experiment with two fine-tuning strategies: "re-training" (initializing with pre-trained weights and starting a new training run) and "fine-tuning" (continuing training the existing model). They found that fine-tuning, which preserves the learning rate and optimizer parameters, yielded better results.
  • Ensembling and Re-ranking: To further boost performance, the authors employ an ensemble of independently trained Transformer models and a language model. They also incorporate a right-to-left re-ranking strategy, a common technique in NMT, to rescore the generated hypotheses.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Training Data:
    • Restricted Track: FCE corpus, NUCLE, Lang-8 Corpus of Learner English, W&I+LOCNESS datasets.
  • Low-Resource Track: A filtered subset of the WikEd corpus.
  • Monolingual Data for Pre-training: 100 million English sentences from the WMT News Crawl corpus.
  • Additional Synthetic Data: 2 million sentences for augmenting training during fine-tuning.
  • Development and Evaluation Data:
    • W&I+LOCNESS Dev set (used for development in both restricted and low-resource tracks).
  • W&I+LOCNESS test set (for official evaluation in the BEA19 shared task).
  • CoNLL 2014 test set.
  • JFLEG test set.

The BEA19 shared task introduced the W&I+LOCNESS dataset, which is notable for its diverse range of English language levels and domains, including texts from native speakers and learners at different proficiency levels.

Key Insights and Ideas

  • Spellchecker Confusion Sets are Powerful for Synthetic Data: The simple, unsupervised method of using spellchecker confusion sets to generate synthetic errors proved surprisingly effective, outperforming more complex methods and significantly boosting performance, especially when genuine annotated data is scarce.
  • Pre-training on Synthetic Data is Crucial: Pre-training the entire Transformer network on this synthetic data, rather than just the decoder, led to substantial improvements, highlighting the benefit of leveraging large amounts of unlabeled data.
  • Fine-tuning Outperforms Re-training: Continuing the training process of the pre-trained model (fine-tuning) was more effective than resetting the training state (re-training) when adapting to the target GEC task.
  • Right-to-Left Re-ranking Benefits High-Recall Systems: The right-to-left re-ranking strategy, while not novel, proved particularly beneficial for their high-recall GEC system, leading to consistent improvements.
  • Ensembling and Re-ranking Yield Significant Gains: Combining multiple models and using re-ranking techniques consistently improved performance across both restricted and low-resource settings.

Key Results

The proposed methods achieved state-of-the-art results:

  • BEA19 Shared Task: The systems ranked first in both the restricted (69.47 F0.5) and low-resource (64.24 F0.5) tracks on the W&I+LOCNESS test set.
  • CoNLL 2014 Test Set: Reported state-of-the-art results of 64.16 M² for the submitted system and 61.30 M² for a system trained on NUCLE and Lang-8 data.
  • Performance on Different Proficiency Levels: The systems showed similar performance on native texts, but the advantage of using real error-annotated data was less pronounced for higher proficiency ESL texts.

Limitations and Future Work

  • Limitations: The authors acknowledge that their synthetic error generation method does not generate context-aware errors. While simple and applicable to any alphabetic language with spell-checkers, it may not fully capture the nuances of human error.
  • Future Work: The paper suggests that further improvements could be achieved by using larger Transformer models. The effectiveness of their approach in low-resource settings also opens avenues for GEC in languages with even less available annotated data. The authors also note that the identical performance on native texts between restricted and low-resource systems warrants further investigation into the role of pre-training.

Summary updated

BibTeX
@inproceedings{grundkiewicz2019neural,
  author = {Roman Grundkiewicz and Marcin Junczys-Dowmunt and Kenneth Heafield},
  title = {{Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure