Minimum Bayes Risk Decoding for Consensus Generation

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2167 papers215 conceptsfrozen 2026-10-04

In traditional natural language generation, we typically ask a model to find the single most likely sequence of words—a strategy known as Maximum A Posteriori (MAP) decoding. Whether using greedy search or beam search, the goal is to find the "mode" of the probability distribution. However, as models grow more complex, researchers have discovered that the most probable sequence is often not the best one. It can be repetitive, overly short, or contain "hallucinations" that are statistically likely but factually incorrect.

Minimum Bayes Risk (MBR) decoding offers a fundamentally different objective: instead of searching for the most probable output, it searches for the most representative output. By generating a pool of candidates and selecting the one that shares the most information with the rest, MBR identifies a "consensus" hypothesis that is robust to the errors of any single sample.

The Failure of the Most Likely Sequence

The core intuition behind MBR, first formalized for word alignment by Bill Byrne and Shankar Kumar, is that the probability distribution of a model contains a vast amount of "dark matter"—high-probability regions that standard search methods ignore.

Standard beam search is prone to picking "safe" but bland sequences. Conversely, random sampling can produce creative but erratic results. Mirac Suzgun and Dan Jurafsky describe MBR as a "wisdom of the crowd" approach. If you sample a model 100 times, the specific hallucinations or errors will likely differ across samples, but the core semantic meaning—the consensus—will remain stable across the majority. MBR selects the candidate that minimizes the "risk" of being wrong relative to this aggregate distribution.

The Mechanism of Consensus

The MBR process follows a simple three-step pipeline:

  1. Sampling: Generate a pool of $N$ candidate hypotheses (e.g., using ancestral sampling or diverse beam search).
  2. Pairwise Comparison: Compare every candidate against every other candidate using a "utility function" $u(h, r)$ that measures similarity.
  3. Selection: Select the candidate $h$ that maximizes the expected utility (the average similarity) across the entire pool.

Mathematically, this identifies the "centroid" of the generated distribution. As Wilker Aziz and Bryan Eikema demonstrate, this approach allows performance to scale with "test-time compute." Unlike beam search, which often plateaus or degrades with very large beam widths, MBR quality consistently improves as you increase the number of samples.

From Token Matching to Neural Metrics

Early MBR systems used simple overlap metrics like BLEU or ROUGE as utility functions. However, these metrics struggle with synonyms and paraphrases. A major breakthrough occurred when researchers began using "Quality-Aware Decoding," replacing token-matching with neural metrics like COMET or BLEURT.

Patrick Fernandes and Graham Neubig showed that using these learnable metrics as the utility function allows MBR to select translations that are more fluent and semantically accurate, even if they have lower raw model probability. This shift has been particularly effective for Large Language Models (LLMs). Andre Martins found that MBR decoding significantly reduces hallucinations in LLM-based translation by forcing the model to stick to the semantic consensus of its own samples.

Beyond Text: Execution-Guided Consensus

The consensus principle extends beyond natural language to structured tasks like code generation and SQL. In these domains, "similarity" isn't just about text overlap; it's about functional behavior.

In Execution-Guided Self-Consistency, researchers like Lukasz Borchmann apply MBR by executing generated code or SQL queries against a database. Instead of comparing the strings of the code, they compare the results of the execution. If five different SQL queries return the exact same table of data, they are considered a consensus group. Selecting a candidate from this group dramatically boosts accuracy, allowing smaller models to rival much larger ones by spending more compute at inference time to find the most robust solution.

Similarly, Daniel D. Johnson and Daniel Tarlow introduced R-U-SURE, which uses MBR to provide uncertainty-aware code suggestions. By maximizing utility across random user intents, the system can identify which parts of a code completion are "stable" across the distribution and which are risky hallucinations, allowing it to truncate suggestions before they become incorrect.

Efficiency and Scaling

The primary drawback of MBR is its quadratic complexity: comparing $N$ samples requires $N^2$ utility calculations. Recent work has focused on making this process more efficient. Yuu Jinnai and Ukyo Honda introduced Model-Based MBR (MBMBR), which uses the model's internal probability estimates to reduce the number of samples needed, and k-medoids MBR to generate diverse yet high-quality outputs.

As the field moves toward "Inference-Time Self-Improvement," as surveyed by Xiangjue Dong, MBR stands as a foundational technique. It transforms the act of generation from a single-path search into a collective deliberation, using the model's own internal distribution to filter out noise and surface the most reliable, high-utility response.

Papers in this version

Showing 60 of 167 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 139 papers
  2. Version 1 · 2026-10-04 · 167 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_minimum_bayes_risk_decoding_d5a70cb9,
  title = {minimum bayes risk decoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minimum-bayes-risk-decoding/rdr_d5a70cb9602841519f97dc188f4f3f9d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_minimum_bayes_risk_decoding_for_consensu_d5a70cb9_v1,
  title = {Minimum Bayes Risk Decoding for Consensus Generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/minimum-bayes-risk-decoding-for-consensus-generation/rdr_d5a70cb9602841519f97dc188f4f3f9d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}