Abstract
This study investigates the relationship between Neural Machine Translation (NMT) quality metrics and human post-editing efficiency. By evaluating 13 NMT systems with professional translators, the research challenges the assumption that incremental improvements in automatic metrics like BLEU linearly translate to faster human editing. Findings suggest that for high-performing models, "better" output does not always equate to "faster" processing, likely due to the cognitive load required to verify highly fluent but potentially inaccurate neural outputs.
Introduction
In the modern translation industry, the workflow has largely transitioned from "translation from scratch" to "post-editing machine translation" (PEMT). The underlying economic assumption is that higher-quality machine output reduces the human effort required to reach a publishable standard. Historically, studies on Phrase-Based MT (PBMT) showed a predictable linear correlation: a 1-point increase in BLEU score typically reduced human editing time by 3–4%.
However, NMT presents a new challenge: "The Fluency Trap." NMT outputs are often remarkably fluent, sounding natural even when they contain factual errors. This study seeks to determine if the traditional "higher BLEU = faster work" rule still applies to high-quality neural models or if the cognitive effort of verification offsets the gains in fluency.
Methodology
The researchers conducted a large-scale experiment involving over 30 professional translators and reviewers focusing on English-to-Czech translation. The study utilized 99 source segments across four domains: News, Legal (Lease), Audit, and Technical documentation.
To create a spectrum of quality, 11 internal NMT models (M01–M11) were developed by checkpointing a state-of-the-art transformer model at various training stages. These were compared against commercial baselines (Google Translate and Microsoft Translator) and two control conditions: translating from scratch ("Source") and editing a human translation ("Reference").
The experiment measured "Think time" (initial cognitive processing) and "Edit time" (active typing), alongside Linguistic Quality Assurance (LQA) to categorize error severity.
Results and Discussion
The study revealed a complex, non-linear relationship between automatic quality metrics and human productivity. While there is a clear correlation between MT quality and the number of manual edits (measured by ChrF similarity), this does not translate directly into time savings.
The BLEU Mirage
For top-performing models, the correlation between BLEU and time spent vanished or even became negative. For the best eight systems, a 1-point BLEU improvement saved only 0.514 seconds per word—significantly less than historical PBMT benchmarks.
The "From Scratch" Break-even Point
Interestingly, translators were often slower when post-editing lower-quality MT systems than when translating from scratch. This indicates a "break-even" quality threshold below which machine translation acts as a cognitive hindrance rather than an aid.
The Revision Paradox
During the second stage of the experiment (revision), higher initial MT quality sometimes led to longer review times. This suggests that when text appears high-quality, reviewers may become more meticulous, spending more time verifying subtle nuances that are harder to spot in fluent NMT output.
Conclusion
The industry must reconsider the expectation that small gains in automatic metrics like BLEU will automatically yield lower costs or faster turnarounds. The "Fluency Trap" of NMT shifts the human burden from "fixing" broken text to "verifying" plausible-sounding output, a task that carries a distinct and often heavier cognitive load. Future research should prioritize measuring this cognitive verification effort over simple edit-distance metrics.
Summary updated