Simulating Realistic Errors for Text Correction and Evaluation

111 concepts77 papers1 proposal

To train a model to evaluate text quality or correct grammatical mistakes, the system must learn to distinguish between well-formed language and flawed text. However, high-quality datasets containing natural human errors or machine translation failures are scarce and expensive to annotate. Historically, researchers bypassed this bottleneck by injecting random noise—such as arbitrary token deletions, insertions, or random word swaps—into clean text corpora.

While scaling random noise is trivial, it introduces a severe distribution shift. Randomly corrupted text looks like robotic nonsense, whereas real human learners and machine translation models make structured, context-dependent mistakes. To bridge this gap, a prominent research direction focuses on linguistically plausible synthetic error generation: using models and structured translation pipelines to simulate the exact types of systematic failures that occur in real-world language processing.

Simulating Errors via Round-Trip Translation

One of the most effective ways to generate realistic errors without relying on manual, language-specific rules is round-trip (or pivot) translation. By translating a clean sentence into an intermediate language and then back into the source language, the translation process naturally introduces subtle semantic shifts, awkward phrasing, and grammatical slips that mimic real-world NLP failures.

Early work by Jey Han Lau et al. in Unsupervised Prediction of Acceptability Judgements demonstrated that round-trip machine translation creates realistic syntactic errors that reflect natural language processing failures, providing an effective unsupervised signal for predicting sentence acceptability. Similarly, Alexandre Berard et al. in LIG-CRIStAL System for the WMT17 Automatic Post-Editing Task leveraged round-trip translation to scale up training data for Automatic Post-Editing (APE), allowing models to learn error-correction patterns when human-annotated data is scarce.

To ensure these errors remain anchored to the desired final output, Diptesh Kanojia et al. in Refer to the Reference: Reference-focused Synthetic Automatic Post-Editing Data Generation proposed a reference-focused approach. Instead of translating the source text forward, they paraphrase and round-trip translate the target reference sentence. This introduces lexical and structural variations that force the translation model to generate plausible errors while maintaining a minimal, human-like edit distance from the gold standard. In low-resource settings, Frank Palma Gomez et al. in Low-Resource Grammatical Error Correction: Selective Data Augmentation with Round-Trip Machine Translation showed that aligning original and round-trip translated tokens allows researchers to extract specific word-level "confusion sets," replicating the contextually incorrect word choices typically made by human language learners.

Exploiting Model Architectures and Inductive Biases

Rather than treating translation systems as black boxes, researchers can exploit the specific architectural flaws and inductive biases of different models to generate targeted error types.

For instance, Wangchunshu Zhou et al. in Improving Grammatical Error Correction with Machine Translation Pairs simulated the grammatical errors of second-language learners by pairing the outputs of two different translation paradigms. By intentionally weakening a phrase-based statistical machine translation (SMT) model (e.g., by reducing its language model weight), they forced it to produce clunky, literal translations. Pairing these literal translations with fluent outputs from a neural machine translation (NMT) model creates a synthetic parallel corpus that captures the "interlanguage" interference common in human language learners.

Similarly, Xin Sun et al. in A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model utilized non-autoregressive translation (NAT) models to synthesize data. Because NAT models predict tokens independently rather than sequentially, they naturally suffer from structural inconsistencies and word repetitions. The authors turned this architectural limitation into a feature, using NAT outputs to train grammatical error correction systems on realistic structural flaws.

Conversely, when modeling machine translation failures for quality estimation, Xiang Geng in NJUNLP's Participation for the WMT2022 Quality Estimation Shared Task demonstrated that autoregressive models are superior to parallel masked language models, as their sequential failure modes more closely mirror the contiguous, multi-word error spans found in real-world translation systems.

Controlled Corruption and Probability-Based Severity

The latest evolution in synthetic error generation moves away from full translation pipelines toward fine-grained, controlled corruption using language models.

In the DirectQE framework, Qu Cui et al. in DirectQE: Direct Pretraining for Machine Translation Quality Estimation generated pseudo-data by masking words in high-quality references and filling them with plausible alternatives from a language model, ensuring the synthetic errors mimic nuanced linguistic mistakes rather than random noise. To prevent the generation of completely ungrammatical text, the authors of Alleviating Distribution Shift in Synthetic Data for Machine Translation Quality Estimation introduced constrained beam search, forcing the generator to produce high-probability variations that remain structurally similar to the reference.

Furthermore, the severity of these simulated errors can be explicitly controlled. Xiang Geng et al. in Unify Word-level and Span-level Tasks: NJUNLP's Participation for the WMT2023 Quality Estimation Shared Task developed a pipeline where replacement tokens are sampled based on their generation probabilities from a language model. Lower-probability tokens serve as a direct proxy for higher error severity, allowing for the automated creation of large-scale datasets for fine-grained, span-level quality estimation. As surveyed by Haofei Zhao et al. in From Handcrafted Features to LLMs: A Brief Survey for Machine Translation Quality Estimation, this ability of large language models to systematically corrupt text across controlled error types has made it possible to distill complex evaluation capabilities into smaller, highly efficient downstream models without the prohibitive cost of human annotation.

Key Foundational Work

Efficient combination of confidence measures for machine translation (2009) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 78 papers
  2. Version 1 · 2026-10-04 · 86 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_synthetic_error_generation_synthetic_err_0d3c78c4,
  title = {synthetic error generation (synthetic error injection)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/synthetic-error-generation-synthetic-error-injection/rdr_0d3c78c41af24269bd70f43953bb4a0d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_synthetic_error_generation_synthetic_err_0d3c78c4_v2,
  title = {synthetic error generation (synthetic error injection)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/synthetic-error-generation-synthetic-error-injection/rdr_0d3c78c41af24269bd70f43953bb4a0d/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake