Artificial Error Generation with Fluency Filtering

Mengyang Qiu, Jungyeul ParkACL 2019

TL;DR

This paper investigates how the "fluency" of artificially generated grammatical errors impacts the performance of Grammatical Error Correction (GEC) models. They found that artificially generated errors with lower fluency significantly improve GEC recall, while higher fluency errors can actually degrade performance by introducing too much noise.

What Problem Does This Paper Address?

Grammatical Error Correction (GEC) is crucial for language learners, aiming to automatically fix errors in text. State-of-the-art GEC systems often use neural machine translation (NMT) models. However, training these models requires vast amounts of high-quality data, which is expensive and time-consuming to create manually. To overcome this, researchers generate artificial errors to augment training data. While these methods boost performance, the quality of these artificial errors is a key question. This paper specifically tackles how the fluency of these generated errors influences GEC model effectiveness.

Methods

The authors build upon the "context window" approach for generating artificial errors. This involves:

  1. Extracting Error Fragments: Identifying error-word/phrase pairs and their surrounding context from existing GEC datasets.
  2. Injecting Errors: Replacing correct fragments in error-free sentences with their incorrect counterparts, ensuring only one error per sentence.
  3. Fluency Filtering: Using a language model (KenLM) to calculate the perplexity (and thus, inverse fluency) of each generated erroneous sentence. They then select sentences based on different fluency levels: highest, median, and lowest.
  4. Training and Evaluation: Training GEC models on datasets augmented with these fluency-filtered artificial errors and evaluating their performance using standard metrics (precision, recall, F0.5).

Datasets

  • Training Data: The paper uses a combination of four datasets from the BEA 2019 Shared Task on GEC: FCE (Train), NUCLE, W&I+LOCNESS (Train), and Lang-8. This amounts to approximately 1.1 million sentence pairs for the baseline model.
  • Artificial Error Generation: The authors applied their error injection procedure to 0.6 million error-free sentences from the training data, resulting in over 0.4 million sentences with injected errors.
  • Evaluation Data: The W&I+LOCNESS development set (4,382 sentences) was used for validation, and the W&I+LOCNESS test set (4,477 sentences) was used for final evaluation.

Key Insights and Ideas

  • Fluency Matters, But Not How You Might Think: The core insight is that artificially generated errors don't need to be perfectly fluent to be effective. In fact, less fluent errors seem to be more beneficial for GEC.
  • Low Fluency Errors Boost Recall: Artificially generated errors with low fluency (high perplexity) are particularly good at improving the recall of GEC models. This suggests they effectively highlight the errors for the model to learn.
  • High Fluency Errors Introduce Noise: Conversely, artificially generated errors that are very close to fluent (low perplexity) can actually hurt performance, especially in terms of precision and overall F0.5 score. This is because the subtle differences might confuse the model.
  • Median Fluency Offers a Balance: The "median fluency" condition strikes a good balance, providing a significant improvement in F0.5 while maintaining reasonable precision.

Key Results

  • The baseline model (trained only on original data) achieved an F0.5 of 43.97%.
  • Models trained with fluency-filtered artificial errors generally outperformed the baseline.
  • The model trained with lowest fluency errors achieved the highest recall (32.96%) and a strong F0.5 of 48.68%.
  • The model trained with median fluency errors achieved the highest F0.5 score (49.03%), demonstrating a good trade-off between precision and recall.
  • The model trained with highest fluency errors performed the worst in terms of F0.5 (42.86%), even worse than the baseline, indicating that highly fluent artificial errors are detrimental.

Limitations and Future Work

  • Single Error per Sentence: The study only injected one artificial error per sentence. Real-world text often contains multiple errors per sentence.
  • Fluency vs. Error Probability: The study focused solely on fluency and did not explore the interplay between fluency and the probability of an error occurring.
  • Future Work: The authors suggest exploring fluency filtering for multi-error sentences and combining fluency filtering with other error generation strategies like machine back-translation to select the best N translations. They also propose investigating the combination of fluency filtering with error probability to create even more effective artificial error data.

Summary updated

BibTeX
@inproceedings{qiu2019artificial,
  author = {Mengyang Qiu and Jungyeul Park},
  title = {{Artificial Error Generation with Fluency Filtering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure