Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data

Zhongtao Liu, Parker Riley, Daniel Deutsch, Alison Lui, Mengmeng Niu, Apurva Shah, Markus Freitag

TL;DR

Collecting gold-standard translation data has historically required costly, time-consuming human translation from scratch. This study evaluates 11 different translation pipelines and shows that human-machine collaboration—either having humans post-edit high-quality machine translation or having LLMs refine human translations—matches or beats human-only translation while cutting costs by up to 40%.

What Problem Does This Paper Address?

High-quality translation datasets are essential for training and evaluating machine translation (MT) systems. Traditionally, the gold standard relies on human linguists translating text from scratch. However, this human-only approach is slow, expensive, and difficult to scale.

While modern large language models (LLMs) and MT systems are often treated as direct competitors to human translators, the authors ask a different question: What happens when humans and machines work together? Specifically, can combining machine generation with human post-editing (or vice versa) produce translation benchmarks that are higher in quality and significantly cheaper than traditional human workflows?

Methods

The authors structure data collection as a two-stage pipeline: Initial Translation followed by Post-Editing (PE). Across these two stages, they test 11 different system combinations:

  1. Initial Translation Sources:
  2. Human: The official WMT23 reference (OrigHumanRef) and a newly commissioned professional human translation (HumanRef).
  3. Machine: The top-performing WMT23 system (BestWMT: GPT-4 5-shot for En $\to$ De; Lan-BridgeMT for Zh $\to$ En) and a middle-tier system (MidWMT: ONLINE-G).
  4. Post-Editing Approaches:
  5. None (raw output).
  6. Human Post-Editing (HumanPE): Professional linguists edit the translation. The authors also tested two independent rounds of human editing (HumanPEx2).
  7. LLM Refinement (LLMRefine): Gemini-1.0 Ultra applied in an iterative error-feedback prompting setup to identify and fix errors.

Evaluation was performed using Multidimensional Quality Metrics (MQM), the gold-standard human evaluation framework where expert raters tag specific error spans and severities (lower MQM scores indicate fewer errors and higher quality).

Datasets

  • WMT23 General MT Shared Task: Evaluated on English $\to$ German (460 paragraph-level segments) and Chinese $\to$ English (1,175 sentence-level segments).
  • Newly Released Benchmark: The authors released a public dataset containing nearly 18,000 translation segments produced across these 11 pipeline combinations, complete with fine-grained MQM human error annotations.

Key Insights and Ideas

  • Complementary Strengths Beat Homogeneous Pipelines: LLMs are exceptionally good at fixing human translations, while humans are best at fixing machine translations:
  • When starting from a human draft (HumanRef), LLM post-editing provided the largest quality gain (cutting major errors by 27.8% on En $\to$ De).
  • When starting from machine output (BestWMT), human post-editing provided the largest gain, whereas LLM refinement actually increased major error counts by 8.5% on En $\to$ De.
  • Familiarity Blindness: Human editors changed far fewer segments when reviewing human translations than when reviewing machine translations. Humans appear prone to overlooking subtle human errors due to natural phrasing, while unnatural machine translation patterns make machine errors immediately salient to human editors.
  • Agreement Scales Inversely with Quality: On poor-quality initial translations (MidWMT), human and LLM post-editors agreed on which segments needed editing 67.4% of the time. On high-quality human text, agreement dropped to 23.9%, showing that obvious blunders generate consensus, but stylistic polishing diverges.

Key Results

  • Collaborative Methods Win Overall: Across both language pairs, the two overall winners landing in the top statistical significance cluster were HumanRef + LLMRefine and BestWMT + HumanPE.
  • On En $\to$ De: HumanRef + LLMRefine achieved an MQM of 2.76 and BestWMT + HumanPE achieved 2.98, both outperforming raw BestWMT (3.30) and raw HumanRef (3.79).
  • On Zh $\to$ En: HumanRef + LLMRefine achieved 1.82 and BestWMT + HumanPE achieved 1.87, beating raw HumanRef (2.05) and raw BestWMT (2.22).
  • Substantial Cost Savings: Assuming industry-standard post-editing costs (~60% of from-scratch translation costs), BestWMT + HumanPE reaches top-tier translation quality at 0.6× the cost of collecting human references from scratch.
  • If human translations already exist, running zero-marginal-cost LLM refinement (HumanRef + LLMRefine) significantly improves quality without needing a second human pass (HumanRef + HumanPEx2), which costs 2.2× as much for negligible quality return.

Limitations and Future Work

  • Limited Languages and Domains: The analysis is restricted to two high-resource language pairs (En-De and Zh-En) using news/general domain WMT datasets; results may differ on low-resource languages or specialized technical domains.
  • Fixed Cost Assumptions: Relative annotation cost was estimated at a flat 60% based on industry heuristics, though actual costs fluctuate across vendors, task difficulties, and languages.
  • Translator Variance: Noticeable baseline quality variance existed between human translator pools (e.g., OrigHumanRef vs. HumanRef), highlighting a need for deeper study into how post-editing influences inter-annotator agreement.

Summary updated

Abstract

Collecting high-quality translations is crucial for the development and evaluation of machine translation systems. However, traditional human-only approaches are costly and slow. This study presents a comprehensive investigation of 11 approaches for acquiring translation data, including human-only, machine-only, and hybrid approaches. Our findings demonstrate that human-machine collaboration can match or even exceed the quality of human-only translations, while being more cost-efficient. Error analysis reveals the complementary strengths between human and machine contributions, highlighting the effectiveness of collaborative methods. Cost analysis further demonstrates the economic benefits of human-machine collaboration methods, with some approaches achieving top-tier quality at around 60% of the cost of traditional methods. We release a publicly available dataset containing nearly 18,000 segments of varying translation quality with corresponding human ratings to facilitate future research.
BibTeX
@inproceedings{liu2024beyond,
  author = {Zhongtao Liu and Parker Riley and Daniel Deutsch and Alison Lui and Mengmeng Niu and Apurva Shah and Markus Freitag},
  title = {{Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data}},
  booktitle = {WMT},
  year = {2024},
  url = {https://openreview.net/forum?id=6FR5W9UiXC},
}

Loading related content...

Figure