Abstract
While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challenge we refer to as metric bias. As MBR decoding aims to produce translations that score highly according to a specific utility metric, this very process makes it impossible to use the same metric for both decoding and evaluation, as improvements might simply be due to reward hacking rather than reflecting real quality improvements. In this work we find that compared to human ratings, neural metrics not only overestimate the quality of MBR decoding when the same metric is used as the utility metric, but they also overestimate the quality of MBR/QE decoding with other neural utility metrics as well. We also show that the metric bias issue can be mitigated by using an ensemble of utility metrics during MBR decoding: human evaluations show that MBR decoding using an ensemble of utility metrics outperforms a single utility metric.
Introduction
In Machine Translation (MT), the standard goal is to find the most likely translation via greedy or beam search. Minimum Bayes Risk (MBR) decoding offers an alternative by sampling multiple candidate translations and selecting the one that maximizes average similarity (utility) to all other candidates. While MBR significantly boosts scores on automatic metrics like COMET or MetricX, it introduces a phenomenon known as metric bias—a form of reward hacking where the system produces translations that "game" the metric without necessarily improving quality for human readers.
The Challenge of Metric Bias
Researchers previously hoped that evaluating MBR outputs with a metric different from the one used for decoding would provide an unbiased assessment. However, this study demonstrates that neural metrics often share similar blind spots, creating a "neural metric echo chamber." When a single neural metric is used for decoding, other neural metrics tend to overestimate the translation quality, failing to provide an independent second opinion.
This bias often manifests as a trade-off between fluency and accuracy. Single-metric MBR frequently selects candidates that are highly fluent but factually inaccurate (hallucinations). Because many sampled candidates may share similar fluent hallucinations, the MBR process "clusters" around these errors, which neural metrics reward but human raters penalize.
Methodology: Ensembling and Efficiency
To mitigate this bias, the authors propose using an ensemble of utility metrics during decoding. By requiring a candidate to satisfy multiple "judges" with different architectures or training data, ensembling acts as a regularizer against reward hacking. Several strategies were tested to combine scores, including:
- rankAvg: Minimizing the average rank of a candidate across all metrics in the ensemble.
- rankMed / rankMax: Using the median or maximum rank to ensure robustness.
To address the $O(n^2)$ computational cost of comparing all samples in MBR, the authors introduced a two-step decoding approach. This involves an initial filtering step using a reference-free Quality Estimation (QE) metric to select the top $N$ candidates, followed by full MBR decoding on only that subset.
Human Evaluation and Results
The study validated these methods using the Multidimensional Quality Metrics (MQM) framework, where professional raters identify specific errors. The evaluation covered 14 language pairs across resource levels, including FLORES-200 and WMT datasets, with translations generated by Gemini 1.0 Pro.
Key findings include:
- Ensembles Outperform Single Metrics: The
rankAvg:noNCensemble (using commercially-friendly metrics) emerged as the top performer, significantly exceeding the quality of greedy decoding in human MQM scores. - Quantifiable Bias: Human evaluations revealed that single-metric MBR was sometimes rated significantly worse than greedy decoding, despite automatic metrics claiming major improvements.
- Efficiency Gains: The two-step QE-filtering approach successfully provided the benefits of MBR without the prohibitive quadratic computational overhead.
- Higher Correlation: Ensembles demonstrated a higher correlation with human judgments (Pearson and Kendall-Tau) than any individual metric alone.
Conclusion
While MBR decoding is a powerful tool for improving MT quality, its reliance on a single utility metric makes it vulnerable to metric bias. Ensembling multiple metrics provides a robust defense against reward hacking, leading to translations that are genuinely preferred by humans. Future work will focus on reducing the computational cost further and developing metrics that are inherently more resilient to adversarial "gaming" during decoding.
Summary updated