Abstract
This paper summarizes the findings of the WMT24 General Machine Translation shared task, a pivotal evaluation in the era of Large Language Models (LLMs). While LLMs like Claude 3.5-Sonnet and GPT-4 have fundamentally shifted the state-of-the-art, the task reveals that machine translation is far from "solved." Challenges persist in handling noisy speech data, social media slang, and maintaining document-level consistency. The report introduces Error Span Annotations (ESA) for more robust human evaluation and highlights the growing issue of "metric-chasing," where systems optimize for automatic scores at the expense of human-perceived quality.
Introduction
For decades, Machine Translation (MT) research focused on sentence-level translation, primarily within the news domain. However, as systems become more sophisticated, these traditional benchmarks no longer capture the nuances of real-world use. The WMT24 organizers identify several critical gaps in current MT evaluation: the need for domain diversity (e.g., handling slang or archaic styles), the impact of modality noise (ASR errors), the importance of context (moving to paragraph and document levels), and the integrity of evaluation metrics as LLMs begin to dominate.
Methodology
The paper introduces several methodological shifts to address the unique challenges of the "LLM era":
- Error Span Annotations (ESA): This new human evaluation protocol sits between simple Direct Assessment (DA) and complex Multidimensional Quality Metrics (MQM). Annotators highlight specific error spans (minor/major) before assigning a final score, making human scores more robust and discriminative. ESA proved highly efficient, producing 37% more statistical clusters than previous methods with half the annotations.
- Document-Level Focus: Source texts were provided as full paragraphs or documents rather than shuffled sentences, forcing systems to maintain stylistic and terminological consistency.
- LLM Benchmarking: The organizers included 8 state-of-the-art LLMs in a standardized 3-shot prompting setup to compare them directly against specialized MT systems.
- AutoRank: To select systems for human evaluation, a combination of reference-free metrics (MetricX and CometKiwi) was used to avoid the bias inherent in comparing systems against a single human reference.
Datasets and Domains
The task covered 11 language pairs across four primary domains: News, Literary (stories from Archive of Our Own), Speech (YouTube transcriptions with ASR noise), and Social (Mastodon content). Additionally, 11 "challenge sets" were designed to test systems on idioms, gender-neutral language, and prompt injection attacks.
Key Results and Insights
The Rise of LLMs and Top Performers
Claude 3.5-Sonnet emerged as the new benchmark leader, outperforming GPT-4 in 9 out of 11 language pairs by producing more natural and accurate translations. Among participating teams, IOL-Research and Unbabel-Tower70B were the most successful, winning in 10 and 8 language pairs respectively. The shift toward decoder-only architectures is evident across the board, with many systems now utilizing paragraph-level context.
The "Metric Gap" and MBR Decoding
A major discovery was the "Metric Gap": systems using Minimum Bayes Risk (MBR) decoding often rank first on automatic metrics but fail to win in human evaluation. These systems essentially "hill-climb" on the flaws of automatic metrics, highlighting the need for more reliable evaluation frameworks that cannot be easily gamed by specific decoding strategies.
Domain-Specific Challenges
While English-to-Spanish is nearing "solved" status with nearly flawless performance, the Speech domain remains the "final boss." Systems struggle to distinguish between actual source content and errors introduced by the ASR process. Furthermore, in 4 out of 11 language pairs, the best machine systems actually outperformed the human "gold standard" references, illustrating the increasing difficulty of sourcing high-quality human data.
Limitations and Future Work
Despite the progress, several challenges remain. LLMs often introduce "chatter" (unwanted conversational filler), which complicates evaluation. The ESA tool's English-only interface created friction for non-English pairs, and some domains like social media threads still lack sufficient context for perfect judgment. Future evaluations must explicitly penalize verbosity and develop better methods to mitigate the hallucination risks unique to LLM-based translation.
Summary updated