Abstract
Tom Kocmi, Ekaterina Artemova, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Konstantin Dranch, Anton Dvorkovich, Sergey Dukanov, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Howard Lakougna, Jessica Lundin, Christof Monz, Kenton Murray, Masaaki Nagata, Stefano Perrella, Lorenzo Proietti, Martin Popel, Maja Popović, Parker Riley, Mariya Shmatova, Steinthór Steingrímsson, Lisa Yankovskaya, Vilém Zouhar. Proceedings of the Tenth Conference on Machine Translation.
2025.
1. Introduction
The machine translation (MT) community has reached a saturation point where traditional benchmarks, often derived from clean sources like news articles, are no longer sufficient to distinguish between top-tier systems. As modern models—particularly Large Language Models (LLMs)—achieve near-perfect scores on standard prose, the field faces a "ceiling effect." When performance scores cluster at the top, identifying true incremental improvements becomes impossible. The WMT25 shared task addresses this by intentionally moving away from "easy" test sets, focusing instead on finding the breaking points of modern systems through non-standard content and difficult source texts.
2. Methodology
The most significant methodological shift in WMT25 is the introduction of Difficulty Sampling. Rather than selecting source documents at random, organizers utilized a regression model named sentinel-src-25 to estimate translation difficulty. Only the most challenging documents were selected for the final evaluation.
Other key methodological choices included:
- Document-level Evaluation: Systems were provided with entire documents to preserve context, a shift designed to leverage the long-context capabilities of LLMs.
- Error Span Annotation (ESA): A granular human evaluation protocol where professional annotators mark specific error spans and their severity, providing a more precise signal than traditional Likert scales.
- Track Differentiation: The task distinguished between Constrained systems (using limited, open-source data and models under 20B parameters) and Unconstrained systems (e.g., GPT-4.1, Gemini 2.5 Pro).
3. Data and Domains
The evaluation spanned 30 language pairs, including new challenging directions like English→Bhojpuri and English→Maasai. The test sets covered six diverse domains: News, Social (Mastodon threads), Speech (YouTube transcripts), Literary (amateur fiction), Education, and Dialogue.
4. Key Results and Insights
4.1 LLM Dominance and the Metric Bias Trap
Gemini 2.5 Pro emerged as the strongest overall system, placing in the top human evaluation cluster for 14 out of 16 evaluated language pairs. The leaderboard was dominated by unconstrained LLMs, including GPT-4.1 and Claude-4. However, a striking "Metric Bias Trap" was observed: systems like Shy-hunyuan-MT dominated automatic rankings (e.g., COMET) but performed significantly worse in human evaluation. This suggests that some models may be over-optimizing for the specific features captured by automatic metrics rather than actual translation quality.
4.2 Domain Challenges
Speech remains the "final boss" of machine translation. Systems struggled with the disfluencies and errors inherent in source ASR text. Conversely, literary text—despite its stylistic complexity—was handled relatively well by modern models. In low-resource settings, such as translation into Maasai, performance remains "largely unusable," with systems frequently hallucinating or defaulting to related languages like Swahili.
5. Discussion and Conclusion
The WMT25 findings indicate that while LLMs have raised the floor for MT quality, they also introduce new evaluation challenges. Human references are no longer always the gold standard, as professional annotators occasionally prefer LLM outputs over human-written translations.
To continue advancing, the community must:
- Improve Metric Reliability: Address the divergence between automatic metrics and human judgment.
- Enhance Multimodality: Better integrate visual and audio context, which remained underutilized this year.
- Target Edge Cases: Move beyond high scores on easy data to focus on the specific scenarios where systems still fail.
The paper concludes that the path forward for MT research lies not in celebrating high scores, but in hunting for the difficult data that reveals remaining system weaknesses.
Summary updated