Mitigating Metric Bias in Minimum Bayes Risk Decoding

Geza Kovacs, Daniel Deutsch, Markus Freitag

Abstract

While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challenge we refer to as metric bias. As MBR decoding aims to produce translations that score highly according to a specific utility metric, this very process makes it impossible to use the same metric for both decoding and evaluation, as improvements might simply be due to reward hacking rather than reflecting real quality improvements. In this work we find that compared to human ratings, neural metrics not only overestimate the quality of MBR decoding when the same metric is used as the utility metric, but they also overestimate the quality of MBR/QE decoding with other neural utility metrics as well. We also show that the metric bias issue can be mitigated by using an ensemble of utility metrics during MBR decoding: human evaluations show that MBR decoding using an ensemble of utility metrics outperforms a single utility metric.

Introduction

In Machine Translation (MT), the standard goal is to find the most likely translation via greedy or beam search. Minimum Bayes Risk (MBR) decoding offers an alternative by sampling multiple candidate translations and selecting the one that maximizes average similarity (utility) to all other candidates. While MBR significantly boosts scores on automatic metrics like COMET or MetricX, it introduces a phenomenon known as metric bias—a form of reward hacking where the system produces translations that "game" the metric without necessarily improving quality for human readers.

The Challenge of Metric Bias

Researchers previously hoped that evaluating MBR outputs with a metric different from the one used for decoding would provide an unbiased assessment. However, this study demonstrates that neural metrics often share similar blind spots, creating a "neural metric echo chamber." When a single neural metric is used for decoding, other neural metrics tend to overestimate the translation quality, failing to provide an independent second opinion.

Heatmap showing the cross-metric bias where MBR decoding with one neural metric inflates scores across other neural metrics.

This bias often manifests as a trade-off between fluency and accuracy. Single-metric MBR frequently selects candidates that are highly fluent but factually inaccurate (hallucinations). Because many sampled candidates may share similar fluent hallucinations, the MBR process "clusters" around these errors, which neural metrics reward but human raters penalize.

Methodology: Ensembling and Efficiency

To mitigate this bias, the authors propose using an ensemble of utility metrics during decoding. By requiring a candidate to satisfy multiple "judges" with different architectures or training data, ensembling acts as a regularizer against reward hacking. Several strategies were tested to combine scores, including:

  • rankAvg: Minimizing the average rank of a candidate across all metrics in the ensemble.
  • rankMed / rankMax: Using the median or maximum rank to ensure robustness.

To address the $O(n^2)$ computational cost of comparing all samples in MBR, the authors introduced a two-step decoding approach. This involves an initial filtering step using a reference-free Quality Estimation (QE) metric to select the top $N$ candidates, followed by full MBR decoding on only that subset.

Human Evaluation and Results

The study validated these methods using the Multidimensional Quality Metrics (MQM) framework, where professional raters identify specific errors. The evaluation covered 14 language pairs across resource levels, including FLORES-200 and WMT datasets, with translations generated by Gemini 1.0 Pro.

Human MQM scores across language pairs, showing that ensemble methods (right columns) consistently outperform single-metric MBR and greedy decoding.

Key findings include:

  • Ensembles Outperform Single Metrics: The rankAvg:noNC ensemble (using commercially-friendly metrics) emerged as the top performer, significantly exceeding the quality of greedy decoding in human MQM scores.
  • Quantifiable Bias: Human evaluations revealed that single-metric MBR was sometimes rated significantly worse than greedy decoding, despite automatic metrics claiming major improvements.
  • Efficiency Gains: The two-step QE-filtering approach successfully provided the benefits of MBR without the prohibitive quadratic computational overhead.
  • Higher Correlation: Ensembles demonstrated a higher correlation with human judgments (Pearson and Kendall-Tau) than any individual metric alone.

Conclusion

While MBR decoding is a powerful tool for improving MT quality, its reliance on a single utility metric makes it vulnerable to metric bias. Ensembling multiple metrics provides a robust defense against reward hacking, leading to translations that are genuinely preferred by humans. Future work will focus on reducing the computational cost further and developing metrics that are inherently more resilient to adversarial "gaming" during decoding.

Summary updated

Abstract

While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challenge we refer to as metric bias. As MBR decoding aims to produce translations that score highly according to a specific utility metric, this very process makes it impossible to use the same metric for both decoding and evaluation, as improvements might simply be due to reward hacking rather than reflecting real quality improvements. In this work we find that compared to human ratings, neural metrics not only overestimate the quality of MBR decoding when the same metric is used as the utility metric, but they also overestimate the quality of MBR/QE decoding with other neural utility metrics as well. We also show that the metric bias issue can be mitigated by using an ensemble of utility metrics during MBR decoding: human evaluations show that MBR decoding using an ensemble of utility metrics outperforms a single utility metric.
BibTeX
@inproceedings{kovacs2024mitigating,
  author = {Geza Kovacs and Daniel Deutsch and Markus Freitag},
  title = {{Mitigating Metric Bias in Minimum Bayes Risk Decoding}},
  booktitle = {WMT},
  year = {2024},
  url = {https://openreview.net/forum?id=DW8mftt6yO},
}
Figure