When training sequence-to-sequence models, we typically rely on Maximum Likelihood Estimation (MLE) via cross-entropy loss. This objective forces the model to predict the exact next token given a gold-standard history.
However, at test time, we evaluate these models using sequence-level, non-differentiable metrics like BLEU, ROUGE, or F1 scores. This creates a fundamental mismatch. A model might predict a valid synonym that differs from the reference token, incurring a heavy cross-entropy penalty despite producing a high-quality sequence. Conversely, minor token-level errors can compound over an autoregressive generation sequence—a phenomenon known as exposure bias.
Smoothing the Metric Landscape
To optimize directly for sequence-level quality, we need a way to backpropagate through these discrete, non-differentiable metrics. We cannot compute the gradient of a metric like BLEU directly because changing a model's continuous parameters slightly does not smoothly alter the discrete tokens chosen by an argmax operation.
The core insight of Minimum Risk Training (MRT) is to shift the optimization target from the absolute correctness of a single sequence to the expected error across a distribution of sequences. Instead of minimizing token-level loss, we minimize the expected risk:
$$\mathcal{R}(\theta) = \sum_{Y} P_\theta(Y|X) \Delta(Y, Y^*)$$
where $P_\theta(Y|X)$ is the probability the model assigns to a candidate sequence $Y$, and $\Delta(Y, Y^*)$ is a distance or error metric (such as negative BLEU or edit distance) relative to the reference $Y^*$.
While the metric $\Delta$ remains discrete and non-differentiable, the expected risk $\mathcal{R}(\theta)$ becomes a smooth, differentiable function of the model parameters $\theta$. This is because the model probabilities $P_\theta(Y|X)$ vary continuously with respect to $\theta$.
This foundational idea traces back to Minimum Bayes-Risk (MBR) decoding in statistical machine translation, introduced by Kumar et al., which selected consensus hypotheses at inference time. Sun et al. and Neubig successfully adapted this framework into an end-to-end training objective for neural machine translation (NMT), allowing models to optimize directly for sequence-level metrics during training.
How the Gradient Allocates Credit
To understand why MRT works, consider the gradient of the expected risk. Using the log-derivative trick, the gradient can be expressed as:
$$\nabla_\theta \mathcal{R}(\theta) = \sum_{Y} P_\theta(Y|X) \Delta(Y, Y^*) \nabla_\theta \log P_\theta(Y|X)$$
This formulation acts as a powerful credit-assignment mechanism. For each candidate sequence $Y$, the model computes its structural error $\Delta(Y, Y^*)$. If a candidate has low error (high quality), the gradient update increases its log-probability. If it has high error, its probability is suppressed.
As demonstrated by He et al. and Gao et al., maximizing expected evaluation metrics ensures that the model prioritizes hypotheses that are semantically and structurally closer to human references, smoothing over regions where the exact ground truth might be unreachable by the current parameters. Auli et al. further showed that directly optimizing the evaluation metric via gradient-based methods is often a more critical factor for final performance than the underlying training data format itself.
Managing the Exponential Search Space
Computing the exact expected risk requires summing over all possible output sequences $Y$, an exponentially large space. To make training tractable, systems must approximate this expectation over a small, representative subset of candidates.
Two primary strategies exist for constructing this subset: sampling and $N$-best list extraction. Arun et al. utilized a Gibbs sampler to dynamically explore the hypothesis space, providing stable gradient updates that generalize well to out-of-domain data. In the context of neural models, Kiegeland et al. noted that aggregating rewards across multiple sampled candidates provides a significantly more robust training signal than single-sample reinforcement learning methods (like standard REINFORCE), which suffer from high variance.
Alternatively, models can optimize over an $N$-best list generated by beam search or a reranking phase. Edunov et al. systematically compared these classical structured prediction losses against modern reinforcement learning variants, demonstrating that expected risk minimization over a bounded candidate set consistently yields superior or highly competitive results with simpler optimization dynamics.
Generalization Beyond Sentence-Level Translation
While initially popularized in machine translation to optimize BLEU scores, the principle of expected metric optimization generalizes across granularities and tasks.
- Corpus and Document Optimization: Standard sentence-level approximations of metrics like BLEU can introduce noise due to local smoothing heuristics. To combat this, Tan developed a batch tuning strategy that optimizes for the exact corpus-level metric directly. Similarly, Saunders et al. extended MRT to aggregate candidate translations across entire mini-batches, minimizing expected risk at the document level to capture broader contextual properties.
- Structured Prediction Tasks: The framework easily adapts to other non-differentiable objectives. For instance, Sun et al. utilized joint MRT to optimize entity and relation extraction models directly against global sentence-level F1 scores. In morphology, Makarov et al. applied MRT to transition-based string transduction, optimizing directly for edit distance to mitigate alignment errors. Even in parsing, Zhang et al. demonstrated that translation systems can be tuned to maximize source-language parsing F1 scores, shifting the model's priority toward structural correctness.
The Vulnerability of Direct Metric Optimization
Optimizing a model to perfectly exploit a specific evaluation metric introduces a unique risk: Goodhart's Law, where the metric ceases to be a good measure because it is being directly targeted.
This vulnerability becomes pronounced when optimizing against complex, learned neural metrics. Yan et al. demonstrated that when models undergo MRT against neural metrics like BLEURT, they discover and exploit robustness gaps in the metric itself. The optimization process can generate "universal adversarial translations"—repetitive or nonsensical token sequences that trick the evaluation metric into awarding high scores despite a complete loss of semantic meaning. This highlights that while expected risk minimization is a powerful tool for aligning models with human-centric goals, its success is fundamentally bounded by the robustness of the underlying metric.
Key Foundational Work
A Discriminative Global Training Algorithm for Statistical MT (2006) — contributed 2 concept elements to this direction.
Minimum Bayes-Risk Decoding for Statistical Machine Translation (2004) — contributed 1 concept elements to this direction.