MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task

Juraj Juraska, Daniel Deutsch, Mara Finkelstein, Markus Freitag

TL;DR

MetricX-24 is Google’s state-of-the-art regression metric for machine translation evaluation, built on a fine-tuned 13B-parameter mT5-XXL backbone. Its primary innovation is a flexible hybrid model capable of both reference-based evaluation and reference-free Quality Estimation (QE) within a single checkpoint. By introducing targeted synthetic training data and jointly training on both Multidimensional Quality Metrics (MQM) and Direct Assessment (DA) human judgments, MetricX-24 cures common metric blind spots and prevents catastrophic forgetting across languages.

What Problem Does This Paper Address?

Modern learned metrics (like BLEURT, COMET, or MetricX) significantly outperform traditional n-gram metrics (like BLEU or ChrF), but they still suffer from several critical vulnerabilities:

  1. Failure mode blind spots: Learned metrics struggle to penalize pathological translations that rarely appear in standard evaluation corpora, such as undertranslation (dropped clauses), duplicated phrases, or fluent hallucinations (sentences that read smoothly but are completely unrelated to the source).
  2. Catastrophic forgetting during fine-tuning: High-quality MQM human ratings—which ground quality in professional error-span annotations—exist for only a few language pairs. When metrics are fine-tuned purely on MQM, their performance collapses on the dozens of other language pairs covered by Direct Assessment (DA) data.
  3. Model fragmentation: Research teams typically have to maintain separate models for reference-based evaluation and reference-free Quality Estimation (QE).

Methods

MetricX-24 addresses these challenges with three key design improvements over MetricX-23:

  • Hybrid Architecture: The model takes input in three flexible formats during training: source + hypothesis, hypothesis + reference, and source + hypothesis + reference. This enables a single model to act as both a QE metric and a reference-based metric, while learning how much to trust the reference when it might be low quality or ambiguous.
  • Targeted Synthetic Data Augmentation: The authors inject synthetic examples designed specifically around known metric failure modes (e.g., gibberish, truncation, sentence duplication, and swapped fluent sentences). A relatively small ratio of synthetic data is blended into both fine-tuning stages.
  • DA and MQM Data Mixing: MetricX fine-tuning proceeds in two stages: first on DA ratings, then on MQM ratings. To stop the model from forgetting the 40+ language pairs seen in Stage 1, the authors mix in raw DA ratings linearly scaled to the 0–25 MQM range (at a 1:4 to 1:10 ratio) during the second stage.
  • Model Scale & Training: The model adapts the encoder-decoder mT5-XXL (13B parameters) into a scalar regression model trained with Adafactor and Mean Squared Error (MSE) loss using 256 TPUs.

Datasets

  • Human Judgments: WMT Metrics data from 2015–2022. The authors filtered out noisy MTurk into-English DA ratings. High-quality MQM data from 2021–2022 covers 4 language pairs (en-de, en-ru, en-zh, zh-en).
  • Evaluation Sets: Primary evaluation relies on WMT23 MQM (en-de, he-en, and zh-en with cleaned references) and WMT22 en-zh news data. Secondary evaluation utilizes the WMT23 DA set across 8 language pairs.
  • New Synthetic Benchmark: The authors introduce a DEMETR-style diagnostic test set generated from WMT23 DA, evaluating pairwise ranking accuracy across 7 specific failure modes (e.g., undertranslation, gibberish, duplication).

Key Insights and Ideas

  • A tiny pinch of synthetic data fixes glaring blind spots: Because human benchmark datasets are dominated by typical MT errors, models never learn to penalize catastrophic failure modes. Adding synthetic negatives at ratios as low as 1:5,000 in Stage 2 boosted undertranslation accuracy from ~58% to over 96%.
  • Co-training preserves multilingual breadth: Mixing linearly transformed DA data into the MQM fine-tuning phase recovered up to 10 points in Pearson's $r$ on broader language pairs without degrading performance on the target MQM pairs.
  • Quality Estimation is closing the gap: Evaluating MetricX-24 in QE mode (without references) outperformed last year's reference-based MetricX-23 model on the WMT23 MQM benchmark.

Key Results

  • MQM Benchmark: On WMT23 MQM, MetricX-24-Hybrid achieves consistent 1–2 point gains in segment-level pairwise accuracy over MetricX-23 across language pairs, and dramatic system-level improvements (e.g., en-zh pairwise accuracy jumped from 74.36% to 85.90%, and up to 88.46% in the standalone model).
  • Synthetic Test Set: MetricX-24 achieved 94%–100% pairwise accuracy on detecting gibberish, fluent hallucinations, undertranslations, and duplications, compared to MetricX-23’s 38.14% on duplication and 57.75% on undertranslation.
  • Hybrid Robustness: On zh-en with flawed original references, MetricX-24-Hybrid scored 0.5% higher segment-level accuracy than the standalone reference-based model, demonstrating its ability to discount low-quality references in favor of the source text.

Limitations and Future Work

  • Punctuation and Reference Sensitivity: The synthetic punctuation set failed to produce clear gains (~82% accuracy), likely because human references often contain punctuation discrepancies, creating noisy training labels.
  • High Compute Footprint: Training a 13B mT5 backbone across 512-token concatenated inputs required 256 TPUs per run, which limited ablation experiments to single random seeds.
  • Reference Matching: Models struggle to award perfect scores to exact reference matches without degrading general scoring behavior, particularly when human references themselves are imperfect.

Summary updated

Abstract

In this paper, we present the MetricX-24 submissions to the WMT24 Metrics Shared Task and provide details on the improvements we made over the previous version of MetricX. Our primary submission is a hybrid reference-based/-free metric, which can score a translation irrespective of whether it is given the source segment, the reference, or both. The metric is trained on previous WMT data in a two-stage fashion, first on the DA ratings only, then on a mixture of MQM and DA ratings. The training set in both stages is augmented with synthetic examples that we created to make the metric more robust to several common failure modes, such as fluent but unrelated translation, or undertranslation. We demonstrate the benefits of the individual modifications via an ablation study, and show a significant performance increase over MetricX-23 on the WMT23 MQM ratings, as well as our new synthetic challenge set.
BibTeX
@inproceedings{juraska2024metricx,
  author = {Juraj Juraska and Daniel Deutsch and Mara Finkelstein and Markus Freitag},
  title = {{MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Task}},
  booktitle = {WMT},
  year = {2024},
  url = {https://openreview.net/forum?id=bUdmMzK3ve},
}

Loading related content...

Figure