Minimum risk training fine-tunes a translation model by sampling several candidate outputs per source sentence, scoring each with a task metric such as BLEU, and shifting probability toward the better candidates. It reliably beats word-by-word likelihood training. It also sometimes collapses: when the metric is a learned scorer, the model can end up producing one generic, high-frequency sentence for every input. In these setups the per-word output parameters are updated by Adam, which scales each parameter's step by its running gradient magnitude. Under likelihood training, those magnitudes are known to track how often each word occurs. We propose that under risk training the frequency that should set a word's step size is different. What matters is how often the word is contested, meaning it appears in some sampled candidates for a sentence but not in others. Only those occurrences tell the model anything about which output is better.
The concrete proposal is a counter-based optimizer for the vocabulary-indexed parameters (input embeddings and output-projection rows). It keeps one integer per word, the contested count, and scales that word's step by the inverse square root of the count. Everything else keeps Adam. The study has two halves. The first is a measurement: during risk training, do Adam's per-row second moments track corpus frequency, contested frequency, or neither? The second is an intervention: does contested-count scaling improve rare-word translation, and does it delay collapse toward high-frequency outputs? If Adam already tracks contested frequency, the method reduces to a memory saving and we will report that outcome as it is.
Core Claim
Three results make the gap concrete. Frequency-aware SGD for Efficient Embedding Learning with Provable Benefits showed that Adam's and AdaGrad's second moments are nearly linear in token frequency under pointwise losses. It then replaced them with integer counters (CF-SGD) at a half to a third of the optimizer memory. That work never touches sequence-level objectives. Classical Structured Prediction Losses for Sequence to Sequence Learning and Minimum Risk Training for Neural Machine Translation established risk training for neural sequence models, but they apply a single global optimizer without regard to vocabulary frequency. BLEURT Has Universal Translations documents collapse toward a high-frequency generic sentence and explicitly leaves open whether a frequency-aware optimizer could resist it.
Our claim is that the risk gradient for a word's row has a specific structure. Suppose the word appears in every candidate for a sentence, as function words usually do. Then the word's contribution to the risk gradient is weighted by the candidates' centered risk values, which sum to zero across the sample. What remains is mostly variation in context, with little signal about which output is better. The informative events are contested occurrences. So we define, for word $v$, the contested count $c_v$: the number of training sentences so far on which $v$ appears in at least one but not all of the $k$ sampled candidates. The update for that word's rows $\theta_v$, with risk gradient $g_v$ and base step size $\eta$, is
$$\theta_v \leftarrow \theta_v - \frac{\eta}{\sqrt{c_v + 1}}\, g_v .$$
The inverse square root follows the FA-SGD analysis, which derives that rate for a coordinate whose gradients arrive at a given rate. We only change which arrivals are counted.
This gives a falsifiable prediction. Frequent, uncontested words receive steps set by their small contested counts. Those words are exactly the ones a collapse would inflate. The prediction is that these steps will be smaller, relative to Adam's normalized steps, than their raw gradient noise would warrant. Rare content words that are contested whenever they appear receive large steps.
Research Program
The first stage is purely diagnostic. It uses a Transformer pretrained with label-smoothed likelihood and then fine-tuned with expected-risk training. We follow Edunov et al.'s recipe: online beam candidates and label-smoothed initialization, which that paper found essential for stability. At checkpoints we log three quantities for every word: Adam's second moment for its output row, its corpus frequency, and its contested count. We then fit rank correlations by frequency bucket.
This stage extends the likelihood-loss finding of Li et al. to risk objectives. It also connects to Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation, which introduced upward-mobility analysis: tracking whether initially low-ranked correct tokens gain probability. That paper found such mobility under risk training but never split it by word frequency. We compute upward mobility separately per frequency bucket, under each optimizer.
The second stage builds the optimizer. Contested counts are updated from the candidate sets that risk training already samples, so the extra compute is a set comparison over $k$ token sets per sentence. The only added state is one integer per vocabulary item. Vocabulary rows use the counter rule; attention and feed-forward weights keep Adam. This split respects the limitation Li et al. state themselves, that their method covers lookup tables only.
The third stage stress-tests collapse. We run risk training with BLEURT as the reward on English-German. This is the setting in which Yan et al. saw output entropy fall from 11.55 (gold references) to 6.58. We then measure whether contested-count scaling delays or prevents that fall.
Baselines
The naive reference point is the likelihood-trained model with no risk fine-tuning. On IWSLT'14 German-English, Classical Structured Prediction Losses for Sequence to Sequence Learning reports 31.78 BLEU for token-level likelihood. We reproduce this checkpoint and use it as the shared starting point for every method.
The strongest existing method is expected-risk fine-tuning with Adam over online beam candidates, for which the same paper reports 32.84 BLEU on IWSLT'14 German-English. For the harder domain-adaptation case, Kiegeland and Kreutzer report risk training moving a WMT15 model from 20.35 to 27.08 BLEU on IWSLT14. We re-run both with the same candidate count, number of updates and hardware as our method. Each optimizer, including ours, gets an identical learning-rate search budget of eight settings.
To count as beating this baseline, our method must do three things across three seeds. It must match or exceed corpus BLEU. It must significantly improve F1 on the rarest third of reference words. And it must not increase the number of optimizer-state floats.
The second competitor is RPROP. A Comparison of Update Strategies for Large-Scale Maximum Expected BLEU Training found this sign-based per-coordinate method beat AdaGrad and SGD for expected-BLEU training. It gained 1.2 BLEU over baseline on IWSLT 2013 German-English, though for a log-linear phrase-based system, not a neural one. We apply RPROP to the vocabulary rows under the same tuning budget.
Two ablations isolate the key ingredient. The first replaces contested counts with corpus counts, which is plain CF-SGD on the risk loss. The second replaces them with counts of every candidate occurrence. If either matches contested counts, the claim about which frequency matters is wrong. A further control permutes contested counts across words within each frequency bucket. This checks that the gains come from per-word information rather than from a coarse frequency-dependent schedule.
Why not just…?
Why not just keep Adam, since it already proxies frequency? This is the most serious threat, and the diagnostic stage exists to settle it. If Adam's row second moments already correlate with contested counts as tightly as Li et al. found for corpus counts, the counter method will match Adam in quality. Its only advantage would then be memory, and we would present it that way. If the correlation is instead with raw frequency or gradient noise, the prediction is a measurable difference in rare-bucket upward mobility. There is also indirect cause for concern about Adam: Understanding the robustness difference between stochastic gradient descent and adaptive gradient methods shows that adaptive methods amplify low-signal directions. Uncontested frequent-word gradients are exactly such directions.
Why not just reweight the loss per token, as Token-level Adaptive Training does? Token-level Adaptive Training for Neural Machine Translation upweights rare target words in the likelihood loss. It keeps a floor weight of 1.0 because discounting frequent words damaged fluency. Reweighting the loss changes the objective being optimized. Our method changes only the step geometry, so the fixed point of risk training stays the same. We will nevertheless run a variant that applies that paper's exponential weights to token log-probabilities inside the risk gradient, with Adam. If it matches us, the simpler loss-side fix wins.
Why not just mix in the likelihood loss to prevent collapse? Yan et al. report that interpolating the likelihood loss at weight 0.6 keeps BLEURT-driven training stable, and this is a strong, cheap fix. We treat it as a baseline for the collapse experiment. We also test it in combination with our method. The question is whether contested-count scaling allows a smaller likelihood weight, and therefore more metric gain, before entropy falls. If the mixture alone dominates at every weight, our method adds nothing for robustness.
Why not just use sharpness-aware minimization, which already balances easy and hard features? Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning shows this balancing for classification. It doubles the forward and backward cost, however, and risk training is already about an order of magnitude costlier than likelihood training because of candidate decoding. We include it at equal wall-clock time, not equal steps.
Proposed Experiments
The main benchmarks are IWSLT'14 German-English with the Edunov setup, and WMT15-to-IWSLT14 domain adaptation with the Kiegeland and Kreutzer setup. Both use BLEU as the reward. The collapse study uses WMT14 English-German with BLEURT as the reward.
The metrics are corpus BLEU and word-level F1 by frequency tercile. We also measure per-tercile upward mobility and the number of unique generated tokens, which Rare Words Degenerate All Words used as a degeneration measure. For the collapse study we track output entropy and the number of steps until entropy falls below a fixed fraction of the gold-reference value.
The decisive checks come in three stages. The correlation study determines whether a separate counter is needed at all. The contested-versus-corpus-versus-all-occurrence ablation determines whether the contested definition is doing the work. The BLEURT run determines whether frequency-aware steps resist the high-frequency collapse that Yan et al. left open.
We cannot report a prior neural result for any frequency-aware optimizer under risk training: no such result is reported on Lacuna. The numbers above are therefore the only external anchors.
References
- Frequency-aware SGD for Efficient Embedding Learning with Provable Benefits
- Classical Structured Prediction Losses for Sequence to Sequence Learning
- Minimum Risk Training for Neural Machine Translation
- BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training
- Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation
- A Comparison of Update Strategies for Large-Scale Maximum Expected BLEU Training
- Understanding the robustness difference between stochastic gradient descent and adaptive gradient methods
- Token-level Adaptive Training for Neural Machine Translation
- Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learning
- Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings