Synthetic Data Generation for Low-resource Grammatical Error Correction with Tagged Corruption Models

Felix Stahlberg, Shankar KumarNAACL 2024

TL;DR

This paper introduces a novel method for generating synthetic training data for grammatical error correction (GEC) in low-resource languages. By adapting "tagged corruption models" and a new multilingual annotation tool, they create a large dataset that significantly improves GEC performance, especially for smaller models and languages with limited existing data.

What Problem Does This Paper Address?

Grammatical Error Correction (GEC) aims to automatically fix errors in text. While neural sequence-to-sequence models have shown promise, they are notoriously data-hungry. Training these models effectively requires large amounts of human-labeled data, which is scarce for many languages beyond English. Existing methods for generating synthetic data often produce errors that are not diverse enough to cover the wide range of mistakes humans make (e.g., grammar, spelling, punctuation). This paper tackles the challenge of creating high-quality, diverse synthetic GEC data for languages with limited resources, making GEC more accessible for a broader linguistic landscape.

Methods

The core of this work is the adaptation of "tagged corruption models." These models are trained to introduce specific types of grammatical errors into clean sentences, guided by an "error tag." For instance, a tag like "NOUN:INFL" might instruct the model to introduce a noun inflection error.

The authors' key innovations include:

  • Multilingual Rule-Based Error Type Annotation: Existing tools like ERRANT are English-specific. The researchers developed a new, language-agnostic annotation tool based on general linguistic rules and multilingual NLP toolkits (SpaCy for part-of-speech tagging and GNU Aspell for spelling correction). This allows them to categorize errors across different languages.
  • LLM-Powered Tagged Corruption: To overcome the scarcity of training data for the corruption models themselves, they leveraged a fine-tuned PaLM 2 foundation model. This powerful LLM was jointly trained on the gold data from all four target languages.
  • Synthetic Data Generation Pipeline: The process involves:
  • Annotating existing gold development sets with error types using their new tool.
  • Computing the distribution of these error tags.
  • Sampling sentences from a large clean text corpus (mC4).
  • Assigning an error tag to each sampled sentence based on the computed distribution.
  • Using the tagged corruption model to generate a corrupted version of the sentence.
  • Filtering the generated parallel data for quality.

Datasets

  • New Synthetic Dataset (PRE): The authors generated a new dataset consisting of 2.5 million examples per language for German, Spanish, Romanian, and Russian. This dataset is referred to as "PRE."
  • Gold Datasets: For training the corruption model and fine-tuning the GEC models, they used existing gold datasets:
  • German: Falko-Merlin corpus
    • Spanish: COWS-L2H corpus
    • Romanian: RONACC corpus
    • Russian: RULEC-GEC corpus
  • CLANG8 Corpus: Used for experiments combining synthetic and external data, this corpus is a re-annotated version of the Lang-8 learner corpus.

Key Insights and Ideas

  • LLMs are Game-Changers for Data Generation: Leveraging a large language model like PaLM 2 significantly mitigates the data scarcity problem when training the tagged corruption models themselves, which is crucial for low-resource languages.
  • Generalizable Annotation is Key: Developing a multilingual, rule-based annotation tool that doesn't rely on language-specific features is essential for scaling synthetic data generation to many languages.
  • Pre-training with Synthetic Data Boosts Fine-tuning: Even though the synthetic dataset alone doesn't always surpass gold-only training, fine-tuning on gold data after pre-training on the synthetic data consistently yields better results. This highlights the value of synthetic data for adapting models to the GEC domain before final refinement.
  • Small Models and Datasets Benefit Most: The improvements from the PRE→GOLD setup are most pronounced for smaller GEC models and languages with smaller gold training sets. This is particularly important for practical GEC applications where low latency is often a requirement.

Key Results

The paper demonstrates consistent gains across all four languages (German, Spanish, Romanian, Russian) when pre-training GEC models on their synthetic PRE dataset before fine-tuning on gold data (PRE→GOLD setup).

  • Significant Improvements: Relative improvements of up to 30% were observed in F0.5-scores compared to training solely on gold data.
  • Model Size Matters: These improvements are more substantial for smaller model sizes (e.g., mT5-base) and languages with less gold data (e.g., Russian).
  • Competitive Performance: The best setups achieved state-of-the-art results on Spanish and Romanian and matched previous best results on German, using a relatively straightforward training pipeline.
  • Combining Data Sources: Pre-training on a mix of their synthetic data and the CLANG8 corpus further improved performance, suggesting that combining different types of synthetic and real-world data can be beneficial.

Limitations and Future Work

  • Tag Set Expressiveness: The current tag set, while multilingual, is coarse-grained and may not capture complex inter-dependencies between error types.
  • Lexical Focus: The automated annotation tool operates at the lexical level, meaning clausal, sentential, or discourse-level errors are not explicitly represented or targeted.
  • Random Tag Assignment: Error tags are assigned randomly to sentences, and it's not always possible to enforce a specific tag on a given sentence (e.g., corrupting a sentence without a conjunction with a 'cconj' tag).
  • Future Work: The authors suggest that further improvements could come from more expressive tag sets, better handling of complex error dependencies, and exploring more sophisticated methods for generating synthetic data. The release of their dataset and annotation tool is intended to facilitate future research in this area.

Summary updated

BibTeX
@inproceedings{stahlberg2024synthetic,
  author = {Felix Stahlberg and Shankar Kumar},
  title = {{Synthetic Data Generation for Low-resource Grammatical Error Correction with Tagged Corruption Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure