Minimum Bayes Risk (MBR) decoding is a way of choosing a translation by sampling many candidates from the model and keeping the one that agrees most, on average, with all the others. Agreement is scored by a learned quality metric such as COMET. In effect, MBR holds a majority vote among the model's own samples. That vote has a known weakness. Translation models are more confident in literal, "translated-sounding" phrasing (translationese) than in natural phrasing: NLLB is more than 3.3 times more perplexed by human Māori than by its own output (The Benefits of Being Uncertain). If the literal phrasing is also the majority among the samples, a majority vote will tend to elect it. The fast approximations that make neural MBR affordable may make this worse. Low-rank completion keeps only the dominant structure of the agreement matrix, and centroid clustering compresses the samples into a few representatives. Both plausibly discard the minority modes where natural phrasings live.
This note proposes to measure that effect and to test a cheap correction. The correction first groups candidates into clusters. It then scores each candidate by a harmonic mean of its agreement with each cluster, so that every mode of the sample distribution gets one vote. The harmonic mean behaves like a soft minimum, and the candidate must not fail badly against any of the modes. The evaluation splits test sets by whether the source sentence was originally written in that language or was itself a translation. It checks two things: whether fast MBR loses quality specifically on original-source inputs, where translationese inflation cannot hide it, and whether harmonic cluster voting recovers that loss at roughly the cost of centroid-based MBR.
Core Claim
Our claim has three parts. First, standard arithmetic MBR and its accelerated variants have been reported only as averages over mixed or forward-only test sets. These averages can hide a systematic preference for translationese outputs, and that preference should show up as a larger gap to natural references on segments where the target-side reference is original text. Second, the cheap fix of swapping the arithmetic mean for a harmonic mean over individual pseudo-references cannot work under the main acceleration method, for a structural reason. Third, a harmonic mean taken over clusters rather than over samples is both cheap and capable of changing which candidate wins.
The structural reason is short enough to state. Trabelsi et al. show that the candidate-by-pseudo-reference utility matrix is nearly rank-1 (Efficient MBR using Low-Rank Matrix Completion). If $u_{ij} = a_i b_j$ exactly, with $a_i$ a per-candidate factor and $b_j$ a per-reference factor, then both the arithmetic and the harmonic average over $j$ rank candidates purely by $a_i$, so the two rules pick the same winner. Harmonic aggregation therefore differs from ordinary MBR only through the residual beyond the first singular component. That residual is exactly what low-rank completion is designed to throw away. A naive "harmonic PMBR" (PMBR is probabilistic MBR, the low-rank completion method) would quietly collapse back to ordinary MBR.
Clustering keeps that residual by construction. Our rule is:
$$\hat{h} = \arg\max_{h \in \mathcal{H}} \left( \frac{1}{k} \sum_{c=1}^{k} \bar{u}(h, c)^{\,p} \right)^{1/p},$$
where $\mathcal{H}$ is the candidate pool, $k$ is the number of $k$-means clusters of candidate embeddings, and $\bar{u}(h,c)$ is the utility of candidate $h$ against the centroid of cluster $c$, clipped to $[\epsilon, 1]$ so the mean stays defined. The exponent $p$ sets how strongly weak agreement is penalized: $p=1$ is an unweighted arithmetic vote over clusters, $p=-1$ is the harmonic mean, and $p \to -\infty$ approaches the minimum. The rule has two ingredients that we ablate separately. Weighting every cluster equally regardless of size is the "one mode, one vote" ingredient. The harmonic exponent is the "do not fail any mode" ingredient. Utility is computed against centroids only, so cost stays at the $O(Nk)$ of centroid-based MBR, where $N$ is the number of candidates.
Research Program
The work has three stages.
The diagnostic stage builds full $N \times N$ COMET utility matrices ($N=1024$, epsilon sampling) for test sets with known source origin. It measures how strongly the first singular value dominates the rest, comparing original-source with translated-source segments. It then runs full MBR, PMBR, centroid-based MBR (CBMBR) and agreement-constrained PMBR (AC-PMBR) on identical candidate pools. Each selected output is scored for adequacy and for naturalness.
The method stage implements the cluster power-mean rule inside the mbrs library (mbrs) and sweeps $p$ and $k$.
The reporting stage applies harmonic aggregation where it belongs: as a summary across the original-source and translated-source partitions. Following How not to Lie with a Benchmark, this exposes a decoder that wins only on the easy, translationese-inflated partition. We always report the per-partition numbers alongside the summary.
The prior work plays three roles here. Three papers motivate the proposal by showing that translationese inflates scores and flips rankings: Zhang and Toral find $R=-0.84$ between system quality and translationese gain and observe PROMT rising from 5th to 1st place on original-source text (The Effect of Translationese in MT Test Sets), and Graham et al. show that parity claims vanish on forward-created data (Statistical Power and Translationese, Translationese in MT Evaluation). None of these papers studies decoding or learned metrics.
Four papers supply the methods we reuse and stress-test: CBMBR, PMBR, AC-PMBR and mbrs. All four report only speed and average-quality results on unpartitioned WMT sets, and all four aggregate arithmetically. CBMBR is closest to this proposal. It already observes that redundant mediocre candidates can outvote a better one when candidates are pooled from several systems. However, it aggregates arithmetically over centroids, with each centroid effectively weighted by the samples it represents, and it never asks about translationese.
Two papers constrain the design. The perplexity paper warns that the MT model's own perplexity rewards translationese, so we cannot use it as a naturalness score. Edunov et al. show that translationese references mislead overlap metrics (On the Evaluation of MT Trained with Back-Translation), so our naturalness evidence must come from original-language references and human judgment, not from BLEU against translated references.
Baselines
The naive reference point is beam search (MAP decoding) with the same model. On WMT22 En-De with M2M100, mbrs reports 83.9 COMET for COMET-utility MBR against 77.3 for beam search (mbrs). We reproduce that setup first to confirm our pipeline matches before any partitioned analysis.
The strongest existing method on quality is full arithmetic MBR at $N=1024$. The strongest methods on the speed-quality trade-off are PMBR, with 16x fewer utility calls and no reported COMET loss on WMT22 En↔De/Ru, and CBMBR, with a 5.7x speedup in utility computation and up to +0.5 COMET in multi-system pools at $k=64$. AC-PMBR is the strongest under extreme budgets, reporting about 2.5% higher XCOMET than PMBR at 1024x reduction on WMT23 En↔De. No reported result found on Lacuna for any MBR variant on original-source versus translated-source partitions. Those numbers are the first thing this project produces.
All decoders share the same candidate pool, the same utility metric (COMET-22) and the same cluster embeddings. We match them on utility-call budget, and we report wall-clock time separately because encoding cost dominates once the pairwise cost is gone. Hyperparameters ($k$, sampling rate, $p$) are tuned on a single development year and frozen for the test years.
Two ablations isolate the key ingredients. The first uses equal cluster weights with $p=1$, which tests whether "one mode, one vote" alone does the work. The second uses size-weighted clusters with $p=-1$, which tests whether the harmonic penalty does the work without re-weighting. A third control applies sample-level harmonic MBR on the full matrix, to confirm the predicted near-collapse to arithmetic MBR. Beating the baselines means two things. Cluster harmonic voting must significantly improve a held-out metric (MetricX, not the COMET used as utility) on original-source segments, with paired bootstrap tests and at least 385 segments per partition, following the power standard of Graham et al. It must also not lose on translated-source segments, and it must run within 1.2x of CBMBR's cost.
Why not just…?
Why not just rerank MBR's top candidates with a naturalness language model? This is the most serious threat. Edunov et al. essentially recommend pairing adequacy with language-model fluency. We include it as a baseline: a linear combination of MBR expected utility and the perplexity of a target-language model trained only on originally written text. Its weight is tuned on the same development set. The comparison is fair because both methods are post-hoc selections over the same pool. If reranking matches cluster harmonic voting on original-source MetricX and on human naturalness judgments, we will report that the cheaper fix suffices. Cluster voting would then be justified only if it avoids the separate model and the fluent-but-wrong outputs that fluency reranking is known to risk. We check that risk with an MQM-style accuracy audit.
Why not just use CBMBR as it is, since it already de-biases redundant candidates? CBMBR's de-biasing was shown for pooling across several systems, and it still lets large clusters dominate an arithmetic vote. The equal-weight, $p=1$ ablation is precisely "CBMBR with the size weighting removed." If that ablation alone captures the whole gain, the harmonic exponent is unnecessary and we will say so.
Why not just evaluate on original-source data and skip harmonic aggregation entirely? For evaluation, the per-partition numbers are indeed what matter, and we always report them. The harmonic summary is only a convenience for ranking. The decoding contribution does not depend on it. Discarding translated-source data is also not available in our low-resource setting: every FLORES-200 non-English source is a translation from English, and halving the data costs statistical power, a cost Graham et al. quantify.
Why not just take the minimum or a low quantile over clusters instead of the harmonic mean? The minimum is the $p \to -\infty$ end of our sweep, so it is tested directly. We expect it to be noisy, because a single bad centroid decides the result, but the sweep settles the question.
Proposed Experiments and Decisive Checks
The data are WMT16–18 newstest with source-origin labels for En↔De, En↔Ru and Zh→En. On translated-source segments, the target-side reference is original text, which gives us natural references. We add the Edunov et al. double-translation bitexts for En↔De/Ru, which separate source-side from reference-side translationese. WMT22/23 En↔De (forward-only) let us reproduce the reported baselines. FLORES-200 English↔Spanish/Japanese/Māori with NLLB-200-3.3B covers the low-resource regime where the perplexity gap is largest. Candidate generation uses M2M100 and NLLB at $N=1024$.
Adequacy is measured with MetricX and XCOMET, held out from the COMET utility to avoid rewarding overfitting to the utility metric. Naturalness is measured by the perplexity of a target-language model trained only on originally written text, by similarity to original-language references, and by a small blind human preference study on original-source segments.
The first decisive check is the residual analysis. If $\sigma_2/\sigma_1$ is negligible on both partitions and every aggregation rule picks the same winner in nearly all segments, the translationese-amplification hypothesis is wrong for current neural utilities. We would publish that negative result and stop the method stage.
The second check is whether fast MBR's quality loss relative to full MBR is concentrated on original-source segments. If the loss is uniform across partitions, existing speed claims transfer safely to natural text, which is itself a useful finding.
The third check is whether cluster harmonic voting beats both the fluency-reranking baseline and the equal-weight ablation on original-source adequacy and naturalness, at CBMBR-level cost. Only a win on all three counts supports the full claim.
References
- The Benefits of Being Uncertain: Perplexity as a Signal for Naturalness in Multilingual Machine Translation
- Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms
- mbrs: A Library for Minimum Bayes Risk Decoding
- How not to Lie with a Benchmark: Rearranging NLP Leaderboards
- The Effect of Translationese in Machine Translation Test Sets
- Statistical Power and Translationese in Machine Translation Evaluation
- Translationese in Machine Translation Evaluation
- Centroid-Based Efficient Minimum Bayes Risk Decoding
- Agreement-Constrained Probabilistic Minimum Bayes Risk Decoding
- On The Evaluation of Machine Translation SystemsTrained With Back-Translation