Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet

Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popovic, Mariya Shmatova, Steinthór Steingrímsson, Vilém Zouhar

Abstract

This paper summarizes the findings of the WMT24 General Machine Translation shared task, a pivotal evaluation in the era of Large Language Models (LLMs). While LLMs like Claude 3.5-Sonnet and GPT-4 have fundamentally shifted the state-of-the-art, the task reveals that machine translation is far from "solved." Challenges persist in handling noisy speech data, social media slang, and maintaining document-level consistency. The report introduces Error Span Annotations (ESA) for more robust human evaluation and highlights the growing issue of "metric-chasing," where systems optimize for automatic scores at the expense of human-perceived quality.

Introduction

For decades, Machine Translation (MT) research focused on sentence-level translation, primarily within the news domain. However, as systems become more sophisticated, these traditional benchmarks no longer capture the nuances of real-world use. The WMT24 organizers identify several critical gaps in current MT evaluation: the need for domain diversity (e.g., handling slang or archaic styles), the impact of modality noise (ASR errors), the importance of context (moving to paragraph and document levels), and the integrity of evaluation metrics as LLMs begin to dominate.

Methodology

The paper introduces several methodological shifts to address the unique challenges of the "LLM era":

  • Error Span Annotations (ESA): This new human evaluation protocol sits between simple Direct Assessment (DA) and complex Multidimensional Quality Metrics (MQM). Annotators highlight specific error spans (minor/major) before assigning a final score, making human scores more robust and discriminative. ESA proved highly efficient, producing 37% more statistical clusters than previous methods with half the annotations.
  • Document-Level Focus: Source texts were provided as full paragraphs or documents rather than shuffled sentences, forcing systems to maintain stylistic and terminological consistency.
  • LLM Benchmarking: The organizers included 8 state-of-the-art LLMs in a standardized 3-shot prompting setup to compare them directly against specialized MT systems.
  • AutoRank: To select systems for human evaluation, a combination of reference-free metrics (MetricX and CometKiwi) was used to avoid the bias inherent in comparing systems against a single human reference.

Datasets and Domains

The task covered 11 language pairs across four primary domains: News, Literary (stories from Archive of Our Own), Speech (YouTube transcriptions with ASR noise), and Social (Mastodon content). Additionally, 11 "challenge sets" were designed to test systems on idioms, gender-neutral language, and prompt injection attacks.

Statistics of the WMT24 test sets across various domains and language pairs.

Key Results and Insights

The Rise of LLMs and Top Performers

Claude 3.5-Sonnet emerged as the new benchmark leader, outperforming GPT-4 in 9 out of 11 language pairs by producing more natural and accurate translations. Among participating teams, IOL-Research and Unbabel-Tower70B were the most successful, winning in 10 and 8 language pairs respectively. The shift toward decoder-only architectures is evident across the board, with many systems now utilizing paragraph-level context.

Overview of participating systems, highlighting the dominance of decoder-only architectures and paragraph-level strategies.

The "Metric Gap" and MBR Decoding

A major discovery was the "Metric Gap": systems using Minimum Bayes Risk (MBR) decoding often rank first on automatic metrics but fail to win in human evaluation. These systems essentially "hill-climb" on the flaws of automatic metrics, highlighting the need for more reliable evaluation frameworks that cannot be easily gamed by specific decoding strategies.

Domain-Specific Challenges

While English-to-Spanish is nearing "solved" status with nearly flawless performance, the Speech domain remains the "final boss." Systems struggle to distinguish between actual source content and errors introduced by the ASR process. Furthermore, in 4 out of 11 language pairs, the best machine systems actually outperformed the human "gold standard" references, illustrating the increasing difficulty of sourcing high-quality human data.

Limitations and Future Work

Despite the progress, several challenges remain. LLMs often introduce "chatter" (unwanted conversational filler), which complicates evaluation. The ESA tool's English-only interface created friction for non-English pairs, and some domains like social media threads still lack sufficient context for perfect judgment. Future evaluations must explicitly penalize verbosity and develop better methods to mitigate the hallucination risks unique to LLM-based translation.

Summary updated

Abstract

This overview paper presents the results of the General Machine Translation Task organised as part of the 2024 Conference on Machine Translation (WMT). In the general MT task, participants were asked to build machine translation systems for any of 11 language pairs, to be evaluated on test sets consisting of three to five different domains. In addition to participating systems, we collected translations from 8 different large language models (LLMs) and 4 online translation providers. We evaluate system outputs with professional human annotators using a new protocol called Error Span Annotations (ESA).
BibTeX
@inproceedings{kocmi2024findings,
  author = {Tom Kocmi and Eleftherios Avramidis and Rachel Bawden and Ondrej Bojar and Anton Dvorkovich and Christian Federmann and Mark Fishel and Markus Freitag and Thamme Gowda and Roman Grundkiewicz and Barry Haddow and Marzena Karpinska and Philipp Koehn and Benjamin Marie and Christof Monz and Kenton Murray and Masaaki Nagata and Martin Popel and Maja Popovic and Mariya Shmatova and Steinthór Steingrímsson and Vilém Zouhar},
  title = {{Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet}},
  booktitle = {WMT},
  year = {2024},
  doi = {10.18653/v1/2024.wmt-1.1},
}
Figure