Distance-Shaped Rewards: Training Translation Models Against a Closed-Form Mahalanobis Surrogate of a Neural Metric

Translation models are trained to predict the next reference word but are judged by sequence-level scores. Minimum risk training (MRT) closes this gap. It samples several candidate translations, scores each with the evaluation metric, and raises the probability of the better ones. When that metric is a learned neural scorer such as BLEURT, the model learns to game it. BLEURT Has Universal Translations showed that MRT against BLEURT collapses English-German output into "universal translations", stock sentences that score well regardless of the source. Output entropy fell from 11.55 for references to 6.58. The only fixes on record are empirical: mixing in token-level likelihood at weight $\lambda=0.6$, or ensembling metrics.

We propose to change the shape of the reward instead of fighting the hacking with loss mixing. We would fit a Mahalanobis distance, a weighted squared distance $d_M(a,b)=(e_a-e_b)^\top M(e_a-e_b)$ between frozen sentence embeddings $e$ with a positive semidefinite weight matrix $M$. The distance is fit in closed form so that it reproduces a neural metric's judgments, and the model is then trained with MRT against this distance. Because $d_M$ is a true (pseudo)metric, it obeys the triangle inequality. One output cannot sit close to many references that are far from each other, so a source-independent universal output cannot score well under it. The weight matrix is solved by one ridge regression and one eigendecomposition. That makes it cheap to refit on the model's own current outputs during training, which keeps the surrogate faithful exactly where the model wanders.

Core Claim

Reward hacking of learned metrics in MRT is partly a property of the reward's functional form. A pairwise regressor like BLEURT can assign high scores to a fixed string for every reference. A distance cannot. The bound we rely on follows directly from the triangle inequality for the square-root distance $\sqrt{d_M}$:

$$\sqrt{d_M(u,r_1)}\le\delta \;\text{and}\; \sqrt{d_M(u,r_2)}\le\delta \;\Longrightarrow\; \sqrt{d_M(r_1,r_2)}\le 2\delta,$$

Here $u$ is a candidate output, $r_1$ and $r_2$ are references for two different sources, and $\delta$ is a score threshold. Any output that scores well across a set of sources forces their references to be mutually close under the same metric. The claim to test is this. MRT against a closed-form distance surrogate of COMET keeps most of the semantic gain that neural-metric MRT is meant to deliver. It does so without the entropy collapse and source-independence documented by Yan et al., and without needing likelihood interpolation as a crutch.

The guarantee has one hole, and we state it plainly. Directions in the nullspace of $M$, or outside the embedding subspace we project onto, are invisible to the distance, so hacks can hide there. Much of the experimental program measures whether models find that hole.

How the surrogate is built

The method borrows from closed-form metric learning. Regressive Virtual Metric Learning replaces iterative pairwise-constraint optimization with one ridge regression to target points. Closed-Form Training of Mahalanobis Distance for Supervised Clustering shows a clustering-risk metric reduces to a pseudoinverse; it trained in 13 seconds where an iterative competitor took 5 days. Neither work touches sequence generation or learned evaluation metrics. We reuse the idea that a Mahalanobis metric can be regressed in closed form, but point it at a different target: scalar metric scores on hypothesis-reference pairs.

Concretely, hypotheses and references are embedded with a frozen multilingual encoder. We use the COMET encoder, whose embedding geometry Centroid-Based Efficient Minimum Bayes Risk Decoding already exploits for clustering at decoding time. Differences are projected onto the top $r$ principal directions; we use $r=64$ as default. We then solve ridge regression for a symmetric $M$ so that $b - z^\top M z$ predicts the COMET score of each pair, where $z$ is the projected embedding difference. That is about $r^2/2$ unknowns. Negative eigenvalues are clipped and a small multiple of the identity is added, so the distance has no nullspace inside the projected space.

Every few hundred MRT steps we sample fresh candidates from the current model, score a subset with real COMET, and refit $M$. Each refit costs seconds. The MRT loss is the standard expected risk over a beam-generated candidate set, following Classical Structured Prediction Losses for Sequence to Sequence Learning, with $\Delta=d_M$.

We do not claim large speedups. The encoder pass dominates COMET's cost and remains. The contribution is the inductive constraint, plus cheap on-policy refitting.

Research Program

The work has three parts.

First, we measure surrogate fidelity. How well does a rank-$r$ distance track COMET and human judgments on WMT pairs, as $r$ varies and as the model's output distribution drifts?

Second, we run the training comparison on two setups. IWSLT'14 German-English is cheap and has a published MRT reference point. The WMT English-German setting of Yan et al. is where universal translations were observed.

Third, we probe adversarially. We actively search for strings that score well under the surrogate across many sources. We compare how much a search budget buys against $d_M$ versus against BLEURT and COMET. We also test whether discovered exploits live in the discarded embedding directions.

Baselines

The naive reference point is the maximum-likelihood model with label smoothing, which also serves as the initialization for all fine-tuning runs. Classical Structured Prediction Losses for Sequence to Sequence Learning reports 31.78 BLEU for token-level training on IWSLT'14 German-English.

The strongest established training baseline is sequence-level expected risk against BLEU from the same paper, which reports 32.84 BLEU on IWSLT'14 German-English. We rerun it with identical candidate generation (beam, same size), steps, and learning-rate search.

The most direct competitor on the hacking question is MRT against BLEURT and against COMET, with and without the likelihood interpolation at $\lambda=0.6$ from BLEURT Has Universal Translations. That paper reports the entropy drop to 6.58 but, as far as the file shows, no COMET score for its regularized model. We found no reported COMET result on Lacuna for MRT-with-COMET, so we reproduce it ourselves under the same compute budget.

Two ablations isolate the key ingredient. The first is a non-metric surrogate: a ridge regression of the same size on the concatenated hypothesis and reference embeddings, fit to the same data. This tests whether the distance form, not the closed-form fitting, prevents hacking. The second is a frozen-$M$ variant that is never refit, which isolates on-policy refitting.

We also report test-time minimum Bayes risk decoding with COMET on the MLE model as a non-training reference. All methods get the same number of COMET calls for fitting or scoring where applicable.

Beating the baselines means two things at once. Our method must reach COMET within a small margin of unregularized COMET-MRT and above BLEU-MRT. It must also hold output entropy and source-dependence near the MLE model's levels, without likelihood interpolation.

Why not just…?

Why not just add the likelihood term, as Yan et al. did? It works, and it is a real threat. Interpolation also pulls the model back toward the reference distribution and may cap metric gains. The file reports $\lambda=0.6$ as the best balance, a tuned constant without a guarantee. We run the interpolated baseline at several $\lambda$ values and plot COMET against entropy. If interpolation dominates our method on that curve, the geometric argument has no practical value, and we will say so.

Why not distill COMET into a small neural reward model? A neural regressor on hypothesis-reference pairs has the same functional freedom as BLEURT, so it can learn universal high scorers. The non-metric ridge ablation is the cheap version of this alternative. A small trained network on the same data is the stronger version, and we include it. The prediction is that both get hacked faster than $d_M$ under the adversarial probe.

Why not just decode with MBR at test time? It leaves the generator unchanged and pays $O(N^2)$ metric calls per sentence. Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms cuts this 16-fold, but it is still a per-sentence decoding cost. Discriminative Reranking for Neural Machine Translation notes that reranking is bounded by candidate-pool quality. We report MBR on top of both the MLE model and our trained model. If training adds nothing over MBR on the MLE model, the training-time claim fails.

Why not just train against BLEU, which has no neural blind spots? BLEU-MRT is safe but rewards surface overlap, the very mismatch neural metrics were built to fix. If our method does not beat BLEU-MRT on held-out semantic metrics, it is not worth the extra machinery.

Proposed Experiments

The decisive checks are four.

First, we evaluate on metrics not used for training: BLEURT, chrF and BLEU, plus a small blind human ranking on WMT English-German. This guards against the surrogate merely agreeing with its own teacher.

Second, we run collapse diagnostics. These are output entropy against the 11.55 reference value, the fraction of identical or near-identical outputs across distinct sources, and how well a source can be identified from its output.

Third, we run the adversarial universal-string search under equal budgets against $d_M$, the non-metric surrogate, BLEURT and COMET. We report the best cross-source score achieved and whether the winning strings exploit directions outside the top-$r$ subspace. We vary $r$ to see whether larger subspaces close the hole.

Fourth, we track surrogate drift: the correlation between $d_M$ and true COMET on current-policy samples over training, for frozen versus refit $M$.

The proposal fails cleanly if any of three things happens: models collapse under $d_M$ just as under BLEURT, the non-metric surrogate proves equally robust, or likelihood interpolation matches us on the quality-entropy curve.