Automatic Correction of Human Translations

Jessy Lin, Geza Kovacs, Aditya Shastry, Joern Wuebker, John DeNero

Abstract

Jessy Lin, Geza Kovacs, Aditya Shastry, Joern Wuebker, John DeNero. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

2022.

Introduction: Moving Beyond Machine Errors

For years, the NLP community has focused on Automatic Post-Editing (APE)—using AI to fix the "disfluent" or "robotic" mistakes made by Machine Translation (MT) systems. However, a massive amount of the world's high-stakes content is still translated by humans. Humans do not make the same mistakes as machines. While an MT system might produce a perfectly grammatical but "hallucinated" sentence, a human is more likely to make a typo, a grammatical slip, or a "bilingual error" (like forgetting to translate a specific technical term).

Existing APE models often fail at correcting human translations because they are tuned to fix machine disfluencies, frequently ignoring the specific, sparse errors humans actually make. This paper introduces Translation Error Correction (TEC) as a distinct research challenge focused on automatically fixing errors in human-generated translations.

Examples of error types found in human translations, ranging from monolingual typos to complex bilingual mismatches.

Methodology: The TEC Model

The authors propose a dual-source encoder-decoder model based on the Transformer architecture. Unlike standard MT or GEC (Grammatical Error Correction) models, the TEC model simultaneously processes the source sentence ($s$) and the human's draft translation ($t$).

Key Architectural Features

  • Dual Inputs: By looking at both the source and the draft, the model can identify bilingual errors that a monolingual grammar checker would miss.
  • Copy Mechanism: To prevent over-editing—a common issue where AI "rewrites" a perfectly good human sentence—the model implements a copy-attention mechanism. This allows it to selectively point to and preserve tokens from the original human draft.
  • Synthetic Pre-training: Since large-scale "human-error" data is scarce, the researchers developed a novel pre-training approach. They "corrupted" clean text using character-level perturbations (insertions, deletions, transpositions, and repetitions) to mimic human typos and slips, rather than the fluency errors typical of MT.
Comparison of model architectures and training data for MT, GEC, APE, and the proposed TEC approach.

The ACED Corpus

To benchmark this new task, the researchers introduced the ACED (Asics, Emerson, DigitalOcean) corpus. This dataset consists of 35,261 English–German sentence triples (source, human draft, and final professional revision) across marketing and technical domains. Unlike APE datasets, where the majority of sentences contain fluency errors, the ACED corpus shows a broad distribution of errors, including typos and terminology mismatches.

Key Insights and Results

The specialized TEC model outperformed standard MT, GEC, and state-of-the-art APE models on all ACED datasets, improving the $F_{0.5}$ score by up to 5.1 points over the next best model.

Human-in-the-Loop Evaluation

A user study with nine professional editors revealed several critical insights:

  • High Acceptance: Editors accepted 79% of the model's suggestions.
  • Efficiency and Quality: Assisted editors produced translations rated as significantly higher quality with less manual typing (measured by character edits).
  • Precision is Paramount: In professional workflows, a "false positive" (a wrong suggestion) is more costly than a "false negative" (a missed error). Editors quickly lose trust in systems that make obvious mistakes, making $F_{0.5}$ (which weights precision more heavily) the ideal metric for this task.

Limitations and Future Work

While the model excels at catching mechanical errors, "bilingual errors"—where the translation doesn't match the source meaning—remain the most difficult to correct. Furthermore, the study noted a "distraction penalty": incorrect suggestions can actually slow down editors as they must mentally process and reject the bad advice. Future research will focus on more sophisticated synthetic data generation and the use of TEC as an instructional tool for junior translators.

Summary updated

BibTeX
@inproceedings{lin2022automatic,
  author = {Jessy Lin and Geza Kovacs and Aditya Shastry and Joern Wuebker and John DeNero},
  title = {{Automatic Correction of Human Translations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=QokhliNDYni},
}
Figure