Generation of Synthetic Error Data of Verb Order Errors for Swedish

Judit Casademont Moner, Elena VolodinaNAACL 2022

TL;DR

This paper tackles the scarcity of Swedish grammatical error data by generating synthetic verb order errors. They developed a rule-based pipeline that mimics real learner errors, creating a valuable dataset for training and evaluating Swedish Grammatical Error Detection and Correction (GED/GEC) systems.

What Problem Does This Paper Address?

Training effective Natural Language Processing (NLP) models, especially for tasks like Grammatical Error Detection and Correction (GED/GEC), requires vast amounts of labeled data. However, for many languages, including Swedish, such datasets are scarce. This is particularly true for specific error types, like verb order errors, which can be complex and vary significantly between learners. Existing resources are often limited in size or don't provide the granular error labeling needed for targeted model development. The authors aim to bridge this gap by creating a synthetic dataset that replicates authentic Swedish verb order errors, making GED/GEC research more feasible for this language.

Methods

The core of this work is a rule-based approach to generate synthetic errors. The process involves:

  1. Analyzing Authentic Errors: They first studied a subset of the SweLL learner corpus and its DaLAJ derivative to identify common patterns in verb order errors, specifically focusing on the placement of finite verbs (S-FinV).
  2. Developing Corruption Rules: Based on linguistic analysis and part-of-speech (POS) tagging, they identified key POS patterns (pronouns, adverbs, nouns, proper names) that frequently precede or follow finite verbs in erroneous sentences. These patterns were then reversed to create rules for corrupting clean sentences.
  3. Seed Data Corruption: A corpus of clean Swedish sentences (COCTAILL) was used as the source. The corruption rules were applied to insert exactly one verb order error per sentence.
  4. Data Preparation: The generated corrupted sentences were paired with their original, correct counterparts. The final dataset was then scrambled and prepared for release.

This rule-based method is highlighted as advantageous because it allows for precise control over the generated errors and the ability to assign error labels, making the synthetic data directly applicable to supervised learning tasks.

Datasets

  • SweLL-gold (Volodina et al., 2019): A collection of 502 manually corrected and tagged learner texts used for analyzing authentic error patterns.
  • DaLAJ (Volodina et al., 2021): A derivative of SweLL-gold, formatted as sentence pairs (original-corrected), where each sentence contains only one error. This was used to identify specific S-FinV error patterns.
  • COCTAILL (Volodina et al., 2014): A corpus of 25,960 clean, textbook-based Swedish sentences used as the "seed" data for generating synthetic errors.
  • FakeDaLAJ (S-FinV): The newly generated synthetic dataset, comprising 31,788 corrupted sentences with S-FinV errors, paired with their correct versions. This dataset is released for public use.

Key Insights and Ideas

  • Mimicking Learner Errors is Key: The study emphasizes that simply introducing random errors is less effective than replicating patterns observed in actual learner writing. This approach leads to more realistic synthetic data.
  • Pronouns Drive Verb Order Errors: The analysis revealed that pronoun-verb order errors are the most prevalent type in Swedish learner data, accounting for a significant portion of the generated synthetic errors.
  • Rule-Based Generation Offers Control: While model-based generation can be faster, the rule-based approach provides fine-grained control over error types and enables direct labeling, which is crucial for supervised learning tasks like GED/GEC.
  • Synthetic Data Improves Model Performance: Preliminary experiments suggest that adding synthetic S-FinV data can tangibly improve the performance of models in error classification tasks.

Key Results

  • A synthetic dataset of 31,788 corrupted sentences with verb order errors (S-FinV) was generated.
  • The distribution of error types in the synthetic data closely mirrors that of authentic learner data:
  • Pronoun-Verb errors: 41.05%
    • Adverb-Verb errors: 31.21%
    • Noun-Verb errors: 25.3%
    • Proper Name-Verb errors: 2.44%
  • A small-scale evaluation of the generated data showed that 90% of sentences were rated as "Good" or "Acceptable" by annotators for their similarity to hypothetical learner errors, indicating high quality.

Limitations and Future Work

  • Time-Consuming Rule Design: Manually studying error patterns and designing corruption rules is time-consuming and can be prone to human bias.
  • Limited Scope: This pilot study focuses solely on verb order errors (S-FinV).
  • Future Work:
    • Releasing cleaned DaLAJ sentences for each error tag to facilitate further synthetic data generation by the community.
  • Exploring guided models for error generation as an alternative to purely manual rule design.
  • Extending the process to other error tags and adapting it for other low-resource languages.
  • Further testing of the generated dataset in GED/GEC tasks.

Summary updated

BibTeX
@inproceedings{moner2022generation,
  author = {Judit Casademont Moner and Elena Volodina},
  title = {{Generation of Synthetic Error Data of Verb Order Errors for Swedish}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure