Metrics as Shards: Group-Sparse Risk Training Against Metric Gaming

When a translation model is fine-tuned to score well on an automatic metric, it often learns to fool that metric instead of translating better. Minimum risk training (MRT) is the standard way to do this fine-tuning: the model samples or beam-searches several candidate translations and shifts probability toward the ones the metric likes. Run against learned neural metrics such as BLEURT or BARTScore, it collapses into "universal translations," fluent high-scoring sentences emitted regardless of the input, or into repeated tokens (BLEURT Has Universal Translations). The standard fix is to mix the risk loss with ordinary likelihood training. That fix is blunt. It holds back every change the risk loss wants to make, the real improvements along with the exploits.

This proposal gives the metric-driven update a narrow, inspectable channel: a layer of millions of sparse indicator features (yes/no flags such as "source word Haus aligned to target word house" or "target contains bigram with a") added to the neural model's beam-search score. That layer is trained with MRT against several metrics at once. The key idea is to treat each metric the way distributed training treats a data shard. A feature survives only if it lowers risk under all metrics jointly, and it is pruned if its gain shows up under only one. The bet is that real quality gains are shared across metrics, while exploits are specific to one metric's blind spots, so requiring agreement across metrics filters out gaming without holding back genuine learning.

Core Claim

Two lines of work supply the pieces. Large sparse-feature training showed that joint $\ell_1/\ell_2$ group selection across data shards keeps features with consistent value and discards features that only memorize one shard. Joint Feature Selection in Distributed Stochastic Learning (IterSelSGD) reached 19.12 BLEU on Europarl/News against 15.39 for small-dev-set tuning. Separately, expected-BLEU training has been shown to work on 3.6M sparse features (Large-scale Expected BLEU Training of Phrase-based Reordering Models).

Our claim is that the shard-consistency principle, applied with metrics as the shards instead of data subsets, is a structural defense against metric gaming. The defense is structural because it comes from the shape of the parameter space rather than from a tuned penalty on how far the model may move. It also makes the defense auditable: the pruned features are a readable list of the exploits each metric invited.

Concretely, give feature $f$ one weight $w_{f,m}$ for each training metric $m \in \{1,\dots,M\}$. Each column $w_{\cdot,m}$ is trained on the risk

$$\mathcal{R}_m(w) = \mathbb{E}_{X}\sum_{Y \in \mathcal{C}(X)} Q_{w}(Y\mid X)\,\big(1-\mathrm{metric}_m(Y, Y^*)\big),$$

where $\mathcal{C}(X)$ is a beam of candidates regenerated online for source sentence $X$, and $Q_w$ is the renormalized distribution over that beam under the combined score. The combined score is the frozen neural model's log-probability plus the sparse features weighted by the metric-averaged weights $\bar w_f = \frac{1}{M}\sum_m w_{f,m}$. The rows are coupled by a group penalty,

$$\lambda \sum_f \big\lVert (w_{f,1},\dots,w_{f,M}) \big\rVert_2 ,$$

which zeros out a feature's entire row unless it pays off jointly across metrics. This follows the IterSelSGD selection step with "shard" reread as "metric." The deployed model decodes with $\bar w$ restricted to the surviving features.

Research Program

The base system is a Transformer trained with standard likelihood on WMT En-De, the setting in which BLEURT Has Universal Translations observed collapse, with IWSLT'14 De-En as a second, smaller language pair. The sparse layer contains four feature families. The first is source-target lexical pairs under the model's attention alignment. The second is target bigram and trigram indicators. The third is repetition-count bins. The fourth is source/target length-ratio bins.

All four families are scored incrementally during beam search, in the style of log-linear NMT-plus-SMT feature combination (Improved Neural Machine Translation with SMT Features). That work showed lexical features can anchor output to the source and cut out-of-vocabulary words from 4.6% to 0.9%. Integrating the features into search matters. A layer that only reranks a fixed beam could not steer search toward exploits, so it would test nothing. A layer inside search can steer it, and so can in principle learn a universal translation.

Candidates are regenerated online at every update, because Classical Structured Prediction Losses for Sequence to Sequence Learning found online beam candidates clearly better than static lists. The same choice removes the main limitation of the Auli et al. expected-BLEU work, whose n-best lists were decoded once offline.

The training metrics are BLEURT, COMET and BARTScore. BERTScore and BLEU are held out entirely and used only for evaluation, along with output entropy and a small human judgment study.

The work proceeds in three stages. First, we reproduce collapse with single-metric MRT on the dense model and on the sparse layer, to confirm that the sparse channel can also be exploited. If it cannot, the experiment is uninformative and we say so. Second, we train the group-sparse multi-metric layer and sweep $\lambda$. Third, we audit the result. We list the highest-norm pruned features per metric and check whether they match the known exploit patterns: generic high-frequency phrases, repeated tokens, and filler n-grams like the "with a" insertions that Tuning Computer Vision Models With Task Rewards saw inflate CIDEr.

The sparse layer is a diagnostic instrument as much as a model. It is not a claim that sparse SMT features beat dense fine-tuning in raw quality. If the principle holds, a later step is to carry the cross-metric agreement test into dense fine-tuning, for example by agreement-masking gradients. That is out of scope here.

Baselines

The naive reference point is the likelihood-trained Transformer decoded with beam search and no metric tuning. Every method must beat it on held-out metrics to count as useful. For the general gain available from risk training, the reference is Classical Structured Prediction Losses for Sequence to Sequence Learning. On IWSLT'14 De-En it reports 32.84 BLEU for expected risk against 31.78 BLEU for token-level likelihood. We rerun that dense risk objective with BLEU as the metric to confirm our pipeline reproduces a gain of that size.

The strongest competitor for the specific question of gaming is the defense from BLEURT Has Universal Translations: MRT mixed with likelihood at $\lambda=0.6$, plus metric ensembling. That paper reports output entropy falling from 11.55 for gold references to 6.58 for the BLEURT-optimized model. No reported result was found on Lacuna for the translation quality of its interpolated or ensembled variants, so we reproduce both ourselves. We run it on the dense model and also with the sparse layer, so that the parameter channel is not confounded with the defense.

Two ablations isolate the key ingredient. In the first, the sparse layer is trained on the averaged metric reward with no group penalty. This tests whether cross-metric agreement matters beyond simply mixing rewards. In the second, the sparse layer is trained on the averaged reward with a plain $\ell_1$ penalty of matched sparsity. This tests whether sparsity alone, without the requirement that metrics agree, does the work.

We also include a discriminative reranker in the style of Discriminative Reranking for Neural Machine Translation, which reports 1.0 to 4.1 BLEU over beam search on WMT'20. We train it on the averaged metric, as a conservative option that cannot collapse search.

All methods share one base checkpoint, the same training sentences, the same beam size and number of candidates per update, and the same number of updates. Each gets the same hyperparameter budget: an equal number of runs over its own strength knob. Model selection for every method uses one fixed rule: best dev BLEU subject to dev entropy staying within 10% of the base model's. Neither held-out neural metric is used for selection.

Beating the Yan et al. defense means higher held-out BERTScore and BLEU at equal or higher entropy, with human raters preferring our outputs or rating them no worse. A result that only raises the training metrics does not count.

Why not just…?

Why not just average the metrics into one reward? This is the most serious threat. Yan et al. already report that ensembling mitigates exploitation. The difference is that an average still rewards a feature that gives a large gain on one metric and is neutral on the others. The group penalty rejects such a feature unless the gain is spread across metrics. Whether that difference matters is an empirical question, and the averaged-reward ablation answers it directly. If the ablation matches our method on held-out metrics, and on the planted-exploit test below, the core claim fails and we will report it that way.

Why not just interpolate with likelihood, as Yan et al. do? Interpolation limits all movement away from the base model equally. It should therefore trade some genuine gain for safety. We test this by tracing each method's curve of held-out metric against drift from the base model (measured as the base model's likelihood of the outputs). Our method should reach higher held-out scores at the same drift. If the two curves coincide, interpolation is the simpler choice and wins.

Why not just rerank n-best lists instead of changing search? A reranker cannot produce universal translations, but it is also bounded by the base model's candidate pool, as the DrNMT authors themselves note. It therefore competes on a different point of the safety–gain trade-off. We include it so readers can see whether in-search training earns its added risk.

Why not just use RAML, which avoids sampling from the model? Reward Augmented Maximum Likelihood samples targets around the reference from a distribution shaped by an edit-distance-like reward. It needs that reward to be analytically samplable, which neural metrics are not. It also gave only 36.50 to 36.91 BLEU on WMT'14 En-Fr. We therefore do not treat it as a competitor on the gaming question.

Decisive Checks

The cleanest test uses a planted blind spot. We define a synthetic training metric equal to COMET plus a bonus for a fixed rare token sequence. We then check whether the group-sparse layer prunes the corresponding n-gram features while the averaged-reward ablation adopts them and emits the sequence. This runs with the ground truth known in advance.

On real metrics, the prediction is that single-metric sparse MRT drives entropy toward the collapse regime Yan et al. describe, while our method keeps entropy near the base model's and still improves held-out BERTScore and BLEU over the base. The audit should show pruned features concentrating on generic, repetitive or source-independent patterns. A human study of a few hundred sentences on WMT En-De checks that gains on held-out metrics are real rather than a shared blind spot.

That last point is the main way the idea can fail: if the metrics share exploits, requiring agreement will not catch them. We measure this by the rate of universal-translation outputs that survive. We also report the converse cost, genuine gains that only one metric recognizes and that the penalty discards, by inspecting high-scoring pruned features that humans rate as helpful.