Training for Grammatical Error Correction Without Human-Annotated L2 Learners’ Corpora

Mikio OdaACL 2023

TL;DR

This paper tackles the challenge of training effective Grammatical Error Correction (GEC) models without relying on large amounts of human-annotated L2 learner data. They propose a novel method to "rebuild" existing synthetic datasets by using a GEC model to generate improved target sentences, showing this approach can lead to better pre-training performance and comparable results to traditional methods.

What Problem Does This Paper Address?

Grammatical Error Correction (GEC) is crucial for helping non-native speakers produce grammatically correct text. Data-driven GEC models, especially sequence-to-sequence models like Transformers, thrive on large, high-quality training data. However, creating extensive human-annotated datasets of L2 learner errors is expensive and time-consuming. While synthetic data generation offers a scalable alternative, existing methods can introduce "inappropriate edits" that don't accurately reflect real L2 learner errors, potentially hindering model performance. This paper aims to bridge this gap by improving the quality and relevance of synthetic training data.

Methods

The core of this paper is a method for rebuilding synthetic parallel training data. Instead of using the original error-free sentences as targets in synthetic datasets, the authors propose using a pre-trained GEC model to generate hypothetical target sentences from the corrupted source sentences. The idea is that an imperfect GEC model, while not perfect, might produce target sentences that are more representative of actual corrections than simply using the original, unedited error-free sentence. This "rebuilt" dataset is then used for pre-training GEC models. They specifically use a GECTOR model (based on RoBERTa) for this rebuilding process.

Datasets

The paper utilizes several datasets:

  • Synthetic Datasets:
    • PIE-9M: A large synthetic dataset commonly used for GEC pre-training.
  • C4-200M: Another large synthetic dataset, also used after being downsized to 9M sentences for comparison.
  • Proposed Datasets (D): The rebuilt synthetic datasets generated using their proposed method from PIE-9M and C4-200M.
  • Random Dataset: A baseline synthetic dataset created by random edits.
  • L2 Learner Corpora (for Stages II & III training):
    • W&I+LOCNESS v2.1: A corpus of learner English with CEFR levels.
  • First Certificate in English (FCE) v.2.1
    • National University of Singapore Corpus of Learner English (NUCLE)
    • Lang-8 Corpus of Learner English
  • Evaluation Datasets:
    • CoNLL2014: A standard benchmark for GEC evaluation.
  • BEA2019: Another standard benchmark for GEC evaluation.

Key Insights and Ideas

  • Paradoxical Improvement: The most surprising finding is that using target sentences generated by an imperfect GEC model (which might still contain errors) can lead to better pre-training performance than using perfectly error-free original target sentences. This suggests that the "errors" in the rebuilt synthetic data might be more representative of real-world GEC challenges.
  • Quality over Perfection: The paper highlights that the "quality" of synthetic data for GEC isn't just about being error-free, but about how well the source-target pairs reflect actual grammatical mistakes and correction patterns of L2 learners.
  • Viability Without Human Data: The proposed method demonstrates that it's possible to achieve practical GEC performance without relying on human-annotated L2 learner corpora for the entire training pipeline, which is a significant step towards more accessible GEC systems.
  • Dataset Characteristics Matter: The effectiveness of the rebuilt data varies between PIE-9M and C4-200M, suggesting that the inherent quality and characteristics of the original synthetic dataset influence how much improvement can be gained.

Key Results

  • Outperforms Original Synthetic Data: Pre-training on the rebuilt synthetic datasets (Proposed) consistently outperformed pre-training on the original synthetic datasets (Original) across both PIE-9M and C4-200M. For example, on the CoNLL2014 test set, the Proposed+BEA2019 model achieved an F0.5 score of 62.8, compared to 62.4 for Original+BEA2019 when using PIE-9M for stage I.
  • Comparable to Full Pipeline: The proposed method, when used for pre-training (Stage I) and then fine-tuned with L2 learner corpora (Stages II & III), achieved performance comparable to conventional full pipeline training. For instance, the Proposed+BEA2019 model achieved an F0.5 of 70.6 on BEA2019, very close to the baseline's 70.5.
  • Effective Without L2 Corpora: Models trained solely on the rebuilt synthetic data (Proposed+PIE-34K and Proposed+C4-34K) achieved strong F0.5 scores (62.9 and 71.5 on CoNLL2014 and BEA2019 respectively), demonstrating the potential to bypass human-annotated L2 data entirely.
  • Reduced Edits: The rebuilt datasets showed a reduction in the average number of edits per sentence compared to the original synthetic datasets, suggesting a more focused and potentially more relevant set of corrections.

Limitations and Future Work

  • Model Imperfection: The paper acknowledges that the GEC model used for rebuilding is imperfect, and its predictions might still contain errors. The exact reason for this paradoxical improvement needs further investigation.
  • Dataset Quality Influence: The effectiveness of the proposed method is shown to be dependent on the quality of the original synthetic dataset, with PIE-9M showing more significant gains than C4-200M.
  • Future Work: The authors plan to investigate further reconfigurations and modifications of synthetic parallel data. They also aim to explore the source-target relationships on training data more deeply to clarify the effectiveness of their method and to understand why imperfect target sentences can be beneficial.

Summary updated

BibTeX
@inproceedings{oda2023training,
  author = {Mikio Oda},
  title = {{Training for Grammatical Error Correction Without Human-Annotated L2 Learners’ Corpora}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure