Mitigating Metric Bias in Minimum Bayes Risk Decoding

Geza Kovacs, Daniel Deutsch, Markus Freitag

TL;DR

Minimum Bayes Risk (MBR) decoding with modern neural metrics (like COMET or MetricX) looks stellar on automated benchmarks, but human evaluations reveal this is largely an illusion caused by "metric bias" (reward hacking). In fact, single-metric MBR often performs worse than standard greedy decoding by trading factual accuracy for superficial fluency. The authors show that ensembling diverse metrics during decoding effectively cancels out individual metric blind spots, yielding substantial and statistically significant quality improvements under human evaluation.

What Problem Does This Paper Address?

In machine translation, Minimum Bayes Risk (MBR) decoding samples multiple candidate translations from an LLM and selects the candidate with the highest expected consensus against all others according to a neural "utility metric."

Because optimizing for a metric naturally inflates its scores, standard practice evaluated MBR outputs using an independent metric (e.g., decoding with MetricX and evaluating with COMET). However, this work reveals a critical flaw: modern neural metrics share correlated inductive biases and failure modes. As a result, evaluating an MBR system with a different neural metric still dramatically overestimates translation quality. When put in front of human raters, single-metric MBR translations often fail to beat simple greedy decoding—and sometimes perform significantly worse.

Methods

The authors evaluate MBR and Quality Estimation (QE) decoding across 12 reference-based metrics (including MetricX-23, XCOMET-XXL/XL, COMET22, BLEURT, and lexical baselines) and 5 QE metrics. To eliminate single-metric bias, they introduce metric ensembling strategies:

  1. Rank Aggregation Ensembling: For each candidate pool, candidates are ranked under multiple metrics ($0$ = best, $127$ = worst). The decoding step selects the candidate minimizing summary statistics across metrics—most notably rankAvg (minimum average rank), as well as median, maximum, and 75th-percentile rank.
  2. Two-Step QE Filtering + MBR: Because running full MBR over large candidate sets with an ensemble is computationally expensive ($O(n^2)$ pairwise evaluations), this approach first applies linear-time QE metric filtering to prune 128 samples down to $N$ candidates (e.g., $N=32$), followed by reference-based MBR on the filtered subset (e.g., noncQE(32)noncnolexMBR).
  3. MQM Human Evaluation: Candidates are scored using fine-grained Multidimensional Quality Metrics (MQM; 0 to 25 error points, lower is better) across fluency, accuracy, and other stylistic errors.
       128 Samples
            │
    [QE Filtering]  ◄── Fast linear-time pruning (e.g., top 32)
            │
      32 Candidates
            │
     [MBR Ensemble] ◄── Quadratic-time consensus over diverse metrics
            │
     Best Translation

Datasets

  • Development: FLORES-200 dev set across 10 diverse language pairs (English to Swahili, Igbo, Hindi, Tamil, Somali, Hausa, Malayalam, Gujarati, Hungarian, and Vietnamese) and WMT2022 (en-zh, zh-en, en-de, de-en).
  • Test: FLORES-200 test set (same 10 pairs) and WMT2023 (en-zh, zh-en, en-de, de-en).
  • Human MQM Annotation: 400 source segments evaluated per language across 6 language pairs representing high- to low-resource settings: English to Hausa, Swahili, Malayalam, Hindi, and German, plus Chinese to English.
  • Generation Model: Gemini 1.0 Pro using 5-shot prompt exemplars and $\epsilon$-sampling ($N = 128$).

Key Insights and Ideas

  • Neural Metrics Collude in Reward Hacking: Neural metrics do not just favor translations decoded with themselves; they systematically inflate scores for translations produced by other neural metrics, even when human raters flag severe translation errors.
  • The Modern LLM Fluency Trap: Past literature reported that single-metric MBR outperformed greedy decoding. The authors hypothesize that past gains were driven by fluency improvements. Because modern LLMs already output highly fluent text under greedy decoding, single-metric MBR now primarily rewards fluent hallucinations that happen to be repeated across pseudoreferences, directly harming factual accuracy.
  • Ensembles Cancel Out Blind Spots: Different metrics make different mistakes. When a candidate translation introduces an inaccurate hallucination, MetricX and XCOMET may overlook it if it sounds natural, but CometKiwi or BLEURT penalize it. Ensembling forces the decoder to pick candidates that satisfy diverse evaluative criteria.

Key Results

  • Human Ratings (MQM Scores, lower is better):
  • Single-Metric Failures: Single-metric MBR and QE failed to reliably outperform greedy decoding (overall MQM: greedy 1.52 vs. MetricX MBR 1.59 and MetricX-QE 1.77, $p < 0.01$). In zh-en, MetricX MBR scored a poor 3.15 vs. greedy 2.56.
  • Ensemble Dominance: The single-step ensemble rankAvg:noNC achieved an MQM of 1.28 ($p < 0.001$ over greedy), winning across almost every language pair.
  • Efficient Two-Step Wins: Pruning with QE before ensembled MBR (noncQE(32)noncnolexMBR) matched the top performance at 1.27 MQM while drastically cutting computational overhead.
  • Both top ensembles outperformed the human reference translations (1.80 MQM).
  • Automated Metric Correlation: Ensembling simulated metrics produced higher Kendall-Tau ($\tau \approx 0.35$ on zh-en) and Pearson correlations with human MQM annotations than individual utility metrics.

Limitations and Future Work

  • Generation Diversity: Experiments relied on a single LLM (Gemini 1.0 Pro), one prompting style (5-shot), and a fixed sample size ($n=128$). Results might shift with smaller open-source models or alternative sampling heuristics.
  • Sentence-Level Constraint: Evaluation was strictly segment-level; context windows and token constraints in COMET (512 tokens) and MetricX (1024 tokens) hinder full document-level MBR.
  • Human Eval Budget: Each segment was evaluated by a single annotator, precluding inter-rater agreement analyses.
  • Open Benchmark Question: Because neural evaluation metrics are susceptible to MBR metric bias, developing automated benchmarks that can reliably evaluate MBR-decoded text without expensive human annotation remains an unsolved problem.

Summary updated

Abstract

While Minimum Bayes Risk (MBR) decoding using metrics such as COMET or MetricX has outperformed traditional decoding methods such as greedy or beam search, it introduces a challenge we refer to as metric bias. As MBR decoding aims to produce translations that score highly according to a specific utility metric, this very process makes it impossible to use the same metric for both decoding and evaluation, as any improvement might simply be due to reward hacking rather than reflecting real quality improvements. In this work we demonstrate that compared to human ratings, neural metrics not only overestimate the quality of MBR decoding when the same metric is used as the utility metric, but they also overestimate the quality of MBR/QE decoding with other neural utility metrics as well. We also show that the metric bias issue can be mitigated by using an ensemble of utility metrics during MBR decoding: human evaluations show that MBR decoding using an ensemble of utility metrics outperforms a single utility metric.
BibTeX
@inproceedings{kovacs2024mitigating,
  author = {Geza Kovacs and Daniel Deutsch and Markus Freitag},
  title = {{Mitigating Metric Bias in Minimum Bayes Risk Decoding}},
  booktitle = {WMT},
  year = {2024},
  url = {https://openreview.net/forum?id=DW8mftt6yO},
}

Loading related content...

Figure