Realistic Synthetic Errors for Translation Quality Tasks

This direction is merged into another direction: synthetic error distribution mismatch. Its last version stays readable and citable here.

Version 1 of 118 papers23 conceptsfrozen 2026-10-04

To build a model that can judge or fix machine translation, you first need a dataset of mistakes. However, high-quality human-annotated data—where a person identifies exactly where a translation went wrong—is incredibly scarce and expensive to produce.

For years, researchers bypassed this bottleneck by creating "synthetic" errors: taking a perfect sentence and randomly swapping words, deleting tokens, or using simple rule-based corruption. But as neural machine translation (NMT) has become more sophisticated, these crude methods have failed. Modern NMT systems don't just produce random noise; they produce "fluent but wrong" text that follows complex linguistic patterns.

A growing body of research argues that the key to improving Translation Quality Estimation (QE) and Automatic Post-Editing (APE) is not just more data, but more realistic synthetic errors that bridge the "distribution shift" between artificial noise and actual model failures.

The Synthetic Gap

The fundamental problem identified by researchers like Shamil Chollampatt and Raymond Hendy Susanto is that large-scale synthetic datasets often fail to replicate the systematic error patterns inherent in real-world NMT. If a model is trained only on random word swaps, it becomes an expert at detecting "jumbled" text, but remains blind to the subtle stylistic inconsistencies or "hallucinations" (where the model adds content not present in the source) that characterize modern NMT.

This discrepancy is particularly acute in low-resource settings. Anushka Singh and Ananya B. Sai have shown that when evaluation metrics are trained on artificial data that doesn't align with real-world linguistic failures, their performance degrades significantly when applied to actual translations in languages like those of the Indian subcontinent.

From Random Noise to Simulated Failure

To close this gap, researchers are moving away from rule-based corruption toward "model-based" error generation. The goal is to use the NMT system's own logic against itself to create "pseudo-translations" that look like real mistakes.

One breakthrough approach is CBSQE (Constrained Beam Search for Quality Estimation), developed by Xiang Geng and Yu Zhang. Instead of injecting noise, they use an MT model’s internal probabilities to guide the generation of pseudo-translations. By forcing the model to choose high-probability but ultimately incorrect tokens, they generate errors that are structurally plausible and fluent—exactly the kind of errors that fool standard QE systems.

Similarly, the DCSQE framework introduced by Xiang Geng and Shujian Huang uses constrained generation and a separate "Annotator" model to group token errors into human-like phrases. This ensures the synthetic data reflects the "chunk-based" nature of human mistakes rather than isolated, random word errors.

Leveraging LLMs as Error Annotators

The rise of Large Language Models (LLMs) has provided a new tool for this direction. Rather than using LLMs to fix text, researchers like Xinyuan Li and Yunshi Lan propose using them as "Type-aware" error generators.

Their framework, TypeDA, uses LLMs to generate specific categories of grammatical errors. By prompting the LLM to simulate a specific human mistake—like a subject-verb agreement error or a subtle tense shift—they create training data that is far more representative of real-world human errors than back-translation or random masking.

Why Realism Matters for Generalization

The intuition behind this shift is simple: a supervised model is only as good as its training distribution. If a QE model only sees "obvious" errors during training, it develops a narrow decision boundary. When it encounters a real-world translation that is 90% correct but contains a critical omission, it lacks the nuance to flag it.

By scaling synthetic data to millions of samples—as seen in the QUAK dataset for Korean-English by Sugyeong Eo and Chanjun Park—and ensuring those samples mimic the "fluency" of NMT, researchers can train smaller, more efficient models. For instance, Qu Cui demonstrated with DirectQE that pretraining on specialized pseudo-data allows a model to outperform much larger architectures like BERT by focusing specifically on the nuances of translation quality from the start.

Ultimately, this research direction suggests that the path to better machine translation isn't just about better generators, but about building "critics" that have been trained on the most realistic simulations of human and machine failure possible.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_realistic_synthetic_errors_for_translati_4af939ed,
  title = {Realistic Synthetic Errors for Translation Quality Tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/realistic-synthetic-errors-for-translation-quality-tasks/rdr_4af939edcf4740a294494584ed1aaba8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_realistic_synthetic_errors_for_translati_4af939ed_v1,
  title = {Realistic Synthetic Errors for Translation Quality Tasks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/realistic-synthetic-errors-for-translation-quality-tasks/rdr_4af939edcf4740a294494584ed1aaba8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}