Automatic Correction of Human Translations

Jessy Lin, Geza Kovacs, Aditya Shastry, Joern Wuebker, John DeNero

TL;DR

This paper introduces Translation Error Correction (TEC), a new task focused on automatically fixing errors in human-generated translations. They release the ACED corpus and demonstrate that TEC requires different approaches than machine translation post-editing, showing that their specialized TEC model significantly improves translation quality and reviewer efficiency.

What Problem Does This Paper Address?

While machine translation (MT) has seen significant advancements, a vast amount of translated content is still produced by humans. These human translations, though generally high-quality, are not error-free. They contain different types of errors (e.g., typos, grammatical slips, inconsistencies) compared to machine-generated translations. Existing systems for improving translations, like Automatic Post-Editing (APE), are designed to fix MT errors, primarily fluency issues. However, there's a gap in addressing the specific error patterns found in human translations. This paper aims to fill this gap by introducing the TEC task and developing models tailored for it.

Methods

The core of the paper involves developing a dual-source encoder-decoder model for TEC. This model is unique because it takes both the original source sentence and the human-generated translation as input, allowing it to leverage information from both. A key innovation is the synthetic data generation strategy for pre-training. Instead of using standard MT or APE data, they generate synthetic errors that mimic the types of mistakes humans make. This pre-training is then followed by fine-tuning on their newly created ACED corpus. They also compare their approach against several baselines, including standard MT, Grammatical Error Correction (GEC), and APE models, to highlight the distinct needs of TEC.

Datasets

The paper introduces the ACED corpus, a new benchmark for TEC. It consists of three English-German datasets (ASICS, EMERSON, DIGITALOCEAN) derived from real-world commercial translation workflows. These datasets contain human translations and their subsequent revisions by professional translators. This contrasts with APE datasets, which typically focus on machine-generated translations. They also use data from the WMT18 and WMT21 APE shared tasks for pre-training and comparison.

Key Insights and Ideas

  • Human errors are different: The analysis of the ACED corpus reveals that human translation errors are more diverse and less focused on fluency issues compared to MT errors. This strongly suggests that dedicated TEC models are needed, rather than simply adapting APE systems.
  • Synthetic data matters: Pre-training on synthetic errors that specifically mimic human error patterns significantly boosts TEC performance, outperforming models trained on general MT or APE data.
  • TEC as human-AI collaboration: The user study demonstrates that TEC systems can act as valuable assistants to professional translators, improving both the quality of the final translation and the efficiency of the review process.
  • Trust is crucial: Reviewers' willingness to accept suggestions is heavily influenced by their trust in the system's reliability. High precision is important, but a system that consistently makes correct suggestions is key to saving time.

Key Results

  • The proposed TEC model, pre-trained on synthetic human-like errors, achieved the best overall F0.5 scores across all ACED datasets, outperforming baselines by up to 5.1 points.
  • In a human-in-the-loop user study with 9 professional translators, the TEC system's suggestions were accepted 79% of the time.
  • Reviewers using the TEC system showed a significant reduction in manual typing effort (insertions + deletions).
  • The quality of reviewed translations was significantly higher when assisted by the TEC system, as judged by an independent evaluator.

Limitations and Future Work

The authors acknowledge that while precision is important for user trust, a high-recall system is also desirable. Balancing these two aspects is crucial for future TEC systems. They suggest future work could involve leveraging recent MT systems for initialization or proposing edits, and developing more sophisticated synthetic data generation techniques. They also propose applying TEC to other scenarios, such as assisting language learners, denoising MT training sets, or cleaning up MT evaluation sets. The paper also highlights the potential of TEC as a testbed for studying human-AI interaction in expert domains.

Summary updated

Abstract

We introduce translation error correction (TEC), the task of automatically correcting human-generated translations. Imperfections in machine translations (MT) have long motivated systems for improving translations post-hoc with automatic post-editing. In contrast, little attention has been devoted to the problem of automatically correcting human translations, despite the intuition that humans make distinct errors that machines would be well-suited to assist with, from typos to inconsistencies in translation conventions. To investigate this, we build and release the Aced corpus with three TEC datasets. We show that human errors in TEC exhibit a more diverse range of errors and far fewer translation fluency errors than the MT errors in automatic post-editing datasets, suggesting the need for dedicated TEC models that are specialized to correct human errors. We show that pre-training instead on synthetic errors based on human errors improves TEC F-score by as much as 5.1 points. We conducted a human-in-the-loop user study with nine professional translation editors and found that the assistance of our TEC system led them to produce significantly higher quality revised translations.
BibTeX
@inproceedings{lin2022automatic,
  author = {Jessy Lin and Geza Kovacs and Aditya Shastry and Joern Wuebker and John DeNero},
  title = {{Automatic Correction of Human Translations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=QokhliNDYni},
}
Figure