TL;DR
Collecting gold-standard translation data has historically required costly, time-consuming human translation from scratch. This study evaluates 11 different translation pipelines and shows that human-machine collaboration—either having humans post-edit high-quality machine translation or having LLMs refine human translations—matches or beats human-only translation while cutting costs by up to 40%.
What Problem Does This Paper Address?
High-quality translation datasets are essential for training and evaluating machine translation (MT) systems. Traditionally, the gold standard relies on human linguists translating text from scratch. However, this human-only approach is slow, expensive, and difficult to scale.
While modern large language models (LLMs) and MT systems are often treated as direct competitors to human translators, the authors ask a different question: What happens when humans and machines work together? Specifically, can combining machine generation with human post-editing (or vice versa) produce translation benchmarks that are higher in quality and significantly cheaper than traditional human workflows?
Methods
The authors structure data collection as a two-stage pipeline: Initial Translation followed by Post-Editing (PE). Across these two stages, they test 11 different system combinations:
- Initial Translation Sources:
- Human: The official WMT23 reference (
OrigHumanRef) and a newly commissioned professional human translation (HumanRef). - Machine: The top-performing WMT23 system (
BestWMT: GPT-4 5-shot for En $\to$ De; Lan-BridgeMT for Zh $\to$ En) and a middle-tier system (MidWMT: ONLINE-G). - Post-Editing Approaches:
- None (raw output).
- Human Post-Editing (
HumanPE): Professional linguists edit the translation. The authors also tested two independent rounds of human editing (HumanPEx2). - LLM Refinement (
LLMRefine): Gemini-1.0 Ultra applied in an iterative error-feedback prompting setup to identify and fix errors.
Evaluation was performed using Multidimensional Quality Metrics (MQM), the gold-standard human evaluation framework where expert raters tag specific error spans and severities (lower MQM scores indicate fewer errors and higher quality).
Datasets
- WMT23 General MT Shared Task: Evaluated on English $\to$ German (460 paragraph-level segments) and Chinese $\to$ English (1,175 sentence-level segments).
- Newly Released Benchmark: The authors released a public dataset containing nearly 18,000 translation segments produced across these 11 pipeline combinations, complete with fine-grained MQM human error annotations.
Key Insights and Ideas
- Complementary Strengths Beat Homogeneous Pipelines: LLMs are exceptionally good at fixing human translations, while humans are best at fixing machine translations:
- When starting from a human draft (
HumanRef), LLM post-editing provided the largest quality gain (cutting major errors by 27.8% on En $\to$ De). - When starting from machine output (
BestWMT), human post-editing provided the largest gain, whereas LLM refinement actually increased major error counts by 8.5% on En $\to$ De. - Familiarity Blindness: Human editors changed far fewer segments when reviewing human translations than when reviewing machine translations. Humans appear prone to overlooking subtle human errors due to natural phrasing, while unnatural machine translation patterns make machine errors immediately salient to human editors.
- Agreement Scales Inversely with Quality: On poor-quality initial translations (
MidWMT), human and LLM post-editors agreed on which segments needed editing 67.4% of the time. On high-quality human text, agreement dropped to 23.9%, showing that obvious blunders generate consensus, but stylistic polishing diverges.
Key Results
- Collaborative Methods Win Overall: Across both language pairs, the two overall winners landing in the top statistical significance cluster were
HumanRef + LLMRefineandBestWMT + HumanPE. - On En $\to$ De:
HumanRef + LLMRefineachieved an MQM of 2.76 andBestWMT + HumanPEachieved 2.98, both outperforming rawBestWMT(3.30) and rawHumanRef(3.79). - On Zh $\to$ En:
HumanRef + LLMRefineachieved 1.82 andBestWMT + HumanPEachieved 1.87, beating rawHumanRef(2.05) and rawBestWMT(2.22). - Substantial Cost Savings: Assuming industry-standard post-editing costs (~60% of from-scratch translation costs),
BestWMT + HumanPEreaches top-tier translation quality at 0.6× the cost of collecting human references from scratch. - If human translations already exist, running zero-marginal-cost LLM refinement (
HumanRef + LLMRefine) significantly improves quality without needing a second human pass (HumanRef + HumanPEx2), which costs 2.2× as much for negligible quality return.
Limitations and Future Work
- Limited Languages and Domains: The analysis is restricted to two high-resource language pairs (En-De and Zh-En) using news/general domain WMT datasets; results may differ on low-resource languages or specialized technical domains.
- Fixed Cost Assumptions: Relative annotation cost was estimated at a flat 60% based on industry heuristics, though actual costs fluctuate across vendors, task difficulties, and languages.
- Translator Variance: Noticeable baseline quality variance existed between human translator pools (e.g.,
OrigHumanRefvs.HumanRef), highlighting a need for deeper study into how post-editing influences inter-annotator agreement.
Summary updated