Neural Machine Translation Quality and Post-Editing Performance

Vilém Zouhar, Martin Popel, Ondrej Bojar, Ales Tamchyna

Abstract

This study investigates the relationship between Neural Machine Translation (NMT) quality metrics and human post-editing efficiency. By evaluating 13 NMT systems with professional translators, the research challenges the assumption that incremental improvements in automatic metrics like BLEU linearly translate to faster human editing. Findings suggest that for high-performing models, "better" output does not always equate to "faster" processing, likely due to the cognitive load required to verify highly fluent but potentially inaccurate neural outputs.

Introduction

In the modern translation industry, the workflow has largely transitioned from "translation from scratch" to "post-editing machine translation" (PEMT). The underlying economic assumption is that higher-quality machine output reduces the human effort required to reach a publishable standard. Historically, studies on Phrase-Based MT (PBMT) showed a predictable linear correlation: a 1-point increase in BLEU score typically reduced human editing time by 3–4%.

However, NMT presents a new challenge: "The Fluency Trap." NMT outputs are often remarkably fluent, sounding natural even when they contain factual errors. This study seeks to determine if the traditional "higher BLEU = faster work" rule still applies to high-quality neural models or if the cognitive effort of verification offsets the gains in fluency.

Methodology

The researchers conducted a large-scale experiment involving over 30 professional translators and reviewers focusing on English-to-Czech translation. The study utilized 99 source segments across four domains: News, Legal (Lease), Audit, and Technical documentation.

To create a spectrum of quality, 11 internal NMT models (M01–M11) were developed by checkpointing a state-of-the-art transformer model at various training stages. These were compared against commercial baselines (Google Translate and Microsoft Translator) and two control conditions: translating from scratch ("Source") and editing a human translation ("Reference").

Performance metrics for the internal NMT models (M01–M11) and commercial baselines.

The experiment measured "Think time" (initial cognitive processing) and "Edit time" (active typing), alongside Linguistic Quality Assurance (LQA) to categorize error severity.

Results and Discussion

The study revealed a complex, non-linear relationship between automatic quality metrics and human productivity. While there is a clear correlation between MT quality and the number of manual edits (measured by ChrF similarity), this does not translate directly into time savings.

Total time per word across different MT models and baselines.

The BLEU Mirage

For top-performing models, the correlation between BLEU and time spent vanished or even became negative. For the best eight systems, a 1-point BLEU improvement saved only 0.514 seconds per word—significantly less than historical PBMT benchmarks.

The "From Scratch" Break-even Point

Interestingly, translators were often slower when post-editing lower-quality MT systems than when translating from scratch. This indicates a "break-even" quality threshold below which machine translation acts as a cognitive hindrance rather than an aid.

The Revision Paradox

During the second stage of the experiment (revision), higher initial MT quality sometimes led to longer review times. This suggests that when text appears high-quality, reviewers may become more meticulous, spending more time verifying subtle nuances that are harder to spot in fluent NMT output.

Conclusion

The industry must reconsider the expectation that small gains in automatic metrics like BLEU will automatically yield lower costs or faster turnarounds. The "Fluency Trap" of NMT shifts the human burden from "fixing" broken text to "verifying" plausible-sounding output, a task that carries a distinct and often heavier cognitive load. Future research should prioritize measuring this cognitive verification effort over simple edit-distance metrics.

Summary updated

Abstract

We test the natural expectation that using MT in professional translation saves human processing time. The last such study was carried out by Sanchez-Torron and Koehn (2016) with phrase-based MT, artificially reducing the translation quality. In contrast, we focus on neural MT (NMT) of high quality, which has become the state-of-the-art approach since then and also got adopted by most translation companies. Through an experimental study involving over 30 professional translators for English -> Czech translation, we examine the relationship between NMT performance and post-editing time and quality. Across all models, we found that better MT systems indeed lead to fewer changes in the sentences in this industry setting. The relation between system quality and post-editing time is however not straightforward and, contrary to the results on phrase-based MT, BLEU is definitely not a stable predictor of the time or final output quality.
BibTeX
@inproceedings{zouhar2021neural,
  author = {Vilém Zouhar and Martin Popel and Ondrej Bojar and Ales Tamchyna},
  title = {{Neural Machine Translation Quality and Post-Editing Performance}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.801},
}
Figure