MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task

Juraj Juraska, Daniel Deutsch, Mara Finkelstein, Markus Freitag

Abstract

In this paper, we present the MetricX-24 submissions to the WMT24 Metrics Shared Task and provide details on the improvements we made over the previous version of MetricX. Our primary submission is a hybrid reference-based/-free metric, which can score a translation irrespective of whether it is given the source segment, the reference, or both. The metric is trained on previous WMT data in a two-stage fashion, first on the DA ratings only, then on a mixture of MQM and DA ratings. The training set in both stages is augmented with synthetic examples that we created to make the metric more robust to several common failure modes, such as fluent but unrelated translation, or undertranslation. We demonstrate the benefits of the individual modifications via an ablation study, and show a significant performance increase over MetricX-23 on the WMT23 MQM ratings, as well as our new synthetic challenge set.

Introduction

Evaluating Machine Translation (MT) quality is a foundational challenge in natural language processing. While traditional metrics like BLEU or ChrF rely on surface-level string matching, they often fail to capture the semantic nuances of modern neural machine translation. Neural-based metrics have emerged as a more accurate alternative, yet they face three persistent hurdles: a heavy dependency on high-quality human references, vulnerability to "fluent but incorrect" failure modes (such as hallucinations or undertranslation), and the tendency to lose performance on low-resource languages when fine-tuned on high-quality human ratings (catastrophic forgetting).

MetricX-24, Google’s latest submission to the WMT 2024 Metrics Shared Task, addresses these issues by utilizing a 13B-parameter mT5 model and a novel hybrid training architecture. This "all-in-one" evaluator is designed to function effectively across reference-based and reference-free scenarios while maintaining high robustness across diverse language pairs.

Methodology

Hybrid Input Architecture

MetricX-24-Hybrid represents a shift toward more versatile evaluation. Unlike previous iterations that required separate models for reference-based evaluation and reference-free Quality Estimation (QE), MetricX-24 is trained on three distinct input formats:

  1. Source + Hypothesis
  2. Hypothesis + Reference
  3. Source + Hypothesis + Reference

By training on these varied combinations, the model learns to dynamically weigh the importance of the source text against the reference, allowing it to remain effective even when human references are unavailable or of poor quality.

Two-Stage Fine-Tuning and Data Mixing

The model is initialized from mT5-XXL and undergoes a rigorous two-stage training process:

  • Stage 1: Broad Coverage. The model is first trained on Direct Assessment (DA) ratings covering over 40 languages. This establishes a broad baseline for translation quality across a wide linguistic spectrum.
  • Stage 2: High-Resolution Refinement. The model is then fine-tuned on Multidimensional Quality Metrics (MQM) ratings, which provide more granular, error-based human feedback.

To combat catastrophic forgetting—where the model loses its ability to score low-resource languages during the MQM phase—the researchers implemented a 1:4 DA-to-MQM mixing ratio. This strategic data mixture ensures the model retains its multi-language breadth while gaining the high-resolution scoring capabilities of MQM.

Synthetic Data Augmentation

To address common neural metric failure modes, the training pipeline includes synthetic "attacks." The authors generated examples of common errors—such as removing punctuation, duplicating text, or inserting unrelated but fluent sentences. By including these synthetic errors at a low ratio (as low as 1:5000), the model is forced to recognize these states as high-error, effectively "patching" blind spots that human-rated datasets often overlook.

Evaluation and Results

Performance Gains

MetricX-24-Hybrid demonstrates significant improvements over its predecessor, MetricX-23. On segment-level pairwise accuracy, the model showed a consistent 1–2 point increase. More notably, on system-level evaluation for the English-to-Chinese (en-zh) task, the model achieved a massive 11.5-point leap in accuracy.

Robustness and Versatility

The impact of synthetic data was most evident in the new DEMETR-style challenge set. The model’s ability to detect "unrelated" or "undertranslated" segments improved from approximately 80% to over 98% accuracy. Furthermore, the hybrid model performs nearly as well as specialized, standalone models, proving that a single architecture can handle both reference-based and reference-free tasks without sacrificing performance.

Discussion and Limitations

While MetricX-24 sets a new state-of-the-art benchmark, some challenges remain. The model still exhibits a slight sensitivity to punctuation, sometimes failing to penalize missing marks as heavily as human raters might. Additionally, the model does not always assign a perfect "zero error" score to hypotheses that exactly match the reference, indicating room for improvement in reference-matching training.

Finally, the computational cost of a 13B-parameter model is substantial. While MetricX-24 is ideal for high-stakes research and offline evaluation, its size makes it less suitable for real-time applications compared to lighter metrics like COMET.

Conclusion

MetricX-24 advances the field of MT evaluation by combining a large-scale hybrid architecture with targeted synthetic training. By addressing the core weaknesses of neural metrics—reference dependency and specific failure modes—it provides a more robust and versatile tool for measuring translation quality across dozens of languages.

Summary updated

Abstract

In this paper, we present the MetricX-24 submissions to the WMT24 Metrics Shared Task and provide details on the improvements we made over the previous version of MetricX. Our primary submission is a hybrid reference-based/-free metric, which can score a translation irrespective of whether it is given the source segment, the reference, or both. The metric is trained on previous WMT data in a two-stage fashion, first on the DA ratings only, then on a mixture of MQM and DA ratings. The training set in both stages is augmented with synthetic examples that we created to make the metric more robust to several common failure modes, such as fluent but unrelated translation, or undertranslation. We demonstrate the benefits of the individual modifications via an ablation study, and show a significant performance increase over MetricX-23 on the WMT23 MQM ratings, as well as our new synthetic challenge set.
BibTeX
@inproceedings{juraska2024metricx,
  author = {Juraj Juraska and Daniel Deutsch and Mara Finkelstein and Markus Freitag},
  title = {{MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task}},
  booktitle = {WMT},
  year = {2024},
  url = {https://openreview.net/forum?id=bUdmMzK3ve},
}
Figure