When training machine learning models for structured prediction tasks—such as machine translation, coreference resolution, or object detection—our ultimate goal is to maximize performance on a specific evaluation metric. For translation, we care about BLEU or METEOR; for coreference resolution, we care about F1-score; for object detection, we care about mean Average Precision (mAP).
Yet, the standard recipe for training these models rarely optimizes for these metrics directly. Instead, models are typically trained using Maximum Likelihood Estimation (MLE) or cross-entropy loss. This creates a fundamental objective-metric misalignment.
To understand why this happens, and how a long line of research has worked to fix it, we need to look at how standard loss functions treat errors versus how humans evaluate success.
The Training-Evaluation Gap
Consider a machine translation model predicting the next word in a sentence where the ground truth is "kitten". Under a standard cross-entropy loss, if the model assigns high probability to "cat", it is penalized just as harshly as if it had predicted "refrigerator". The loss function only cares about exact token matches; it is completely blind to semantic or structural proximity.
At test time, however, the model is judged by metrics like BLEU or human evaluation, which recognize that "cat" preserves the meaning of the sentence far better than "refrigerator". As noted by Fırat et al. in their work on Nematus, minimizing cross-entropy does not guarantee an improvement in human-centric quality scores. Nakatani et al. similarly observe in Comparing BERT-based Reward Functions for Deep Reinforcement Learning in Machine Translation that token-level training often yields grammatically correct but semantically awkward outputs because the training objective is fundamentally misaligned with the evaluation criteria.
If our true goal is to maximize an evaluation metric, why not use that metric directly as our loss function?
The Differentiability Barrier
The reason we cannot simply plug BLEU or F1-score into gradient descent is that these metrics are non-differentiable. They rely on discrete operations:
- Taking the
argmaxover a vocabulary to generate a sequence of tokens or bounding boxes. - Counting discrete matches (such as $n$-gram overlaps or true positives).
Because these operations are step functions, their derivatives are zero almost everywhere. If you tweak a model's weights by a tiny fraction ($10^{-6}$), the discrete output sequence remains exactly the same. The metric score does not change, providing a gradient of zero, which leaves gradient descent completely paralyzed.
Early statistical machine translation systems bypassed this by avoiding gradient descent altogether. In a foundational paper, Franz Josef Och introduced Minimum Error Rate Training (MERT), a derivative-free optimization algorithm. MERT works by computing an $N$-best list of candidate translations and performing a line search directly over the non-differentiable metric surface to find the optimal parameter weights.
[Model Parameters] ──> [Generate N-Best List] ──> [Line Search on BLEU Surface] ──> [Update Parameters]
However, the error surfaces of these metrics are notoriously "jagged" and non-convex. To navigate this messy landscape, David A. Smith et al. proposed Minimum Risk Annealing. By applying deterministic annealing, the authors smoothed the optimization landscape early in training and gradually restored its complexity, helping the model escape poor local optima and converge on parameters that directly improved translation quality.
Smoothing via Expected Risk Minimization
As models grew from simple log-linear combinations to complex graphical models and deep neural networks, derivative-free line searches over $N$-best lists became computationally intractable. The field needed a way to make direct metric optimization compatible with gradient-based learning.
The breakthrough intuition was to shift from optimizing the metric of the single, discrete argmax output to optimizing the expected metric over the model's entire output probability distribution.
Instead of evaluating a single sequence, we define an objective based on Expected Risk:
$$\mathcal{L}_{\text{risk}}(\theta) = \sum_{y \in \mathcal{Y}} P(y|x; \theta) \cdot \Delta(y, y^*)$$
where $P(y|x; \theta)$ is the probability the model assigns to an output sequence $y$, and $\Delta(y, y^*)$ is a discrepancy function or task metric (such as $1 - \text{BLEU}$) comparing $y$ to the ground truth $y^*$.
Because the probability $P(y|x; \theta)$ is a smooth, continuous function of the model's parameters $\theta$, we can compute its gradient. We are no longer differentiating the metric itself; we are differentiating the probabilities assigned to different metric scores.
This framework allowed researchers to inject task-specific metrics directly into training across various architectures:
- Conditional Random Fields (CRFs): Yejin Choi et al. introduced Biased Potential Functions in Structured Local Training and Biased Potential Functions for Conditional Random Fields with Application to Coreference Resolution, tuning CRFs directly toward maximizing F1-scores rather than data likelihood.
- Stochastic Learning: Michael L. Wick et al. developed SampleRank, which embeds parameter updates directly into the steps of an MCMC inference chain by enforcing ranking constraints based on domain-specific metrics.
- Machine Translation Reordering: Michael Auli et al. utilized this approach in Large-scale Expected BLEU Training of Phrase-based Reordering Models to optimize translation reordering models directly for expected BLEU scores.
Scaling to Deep Neural Networks
In modern deep learning, calculating the exact sum over all possible sequences $\mathcal{Y}$ is impossible because the sequence space grows exponentially with sentence length. To scale expected risk minimization to deep neural networks, researchers rely on smart sampling and reward distribution adjustments.
One approach is to approximate the full expectation by sampling a small, representative subset of candidate outputs. Danielle Saunders et al. extended this technique in Using Context in Neural Machine Translation Training Objectives by grouping sentences into "pseudo-documents" within a training batch. This allowed the model to approximate and optimize document-level metrics directly, bridging the gap between sentence-level loss functions and global evaluation criteria.
An alternative to calculating expected risk over samples is to modify the target distribution itself so that it inherently reflects the evaluation metric. Mohammad Norouzi et al. introduced this concept with Reward Augmented Maximum Likelihood (RAML).
Standard MLE: Train model to assign 100% probability to the exact ground truth.
RAML Approach: Train model to assign probability proportional to a candidate's task reward.
Instead of forcing the model to only imitate the exact training tokens, RAML samples alternative target sequences from a stationary distribution that is exponentially proportional to their task reward (e.g., BLEU score or edit distance). By training the model to assign higher probability to high-reward sequences and lower probability to low-reward sequences, RAML smoothly integrates the evaluation metric into a standard maximum likelihood framework without requiring complex reinforcement learning policy gradients.
Beyond Text: Generalizing to Task Rewards
While direct metric optimization originated primarily within statistical and neural machine translation, the underlying problem—and its solutions—applies universally across machine learning.
Recently, André Susano Pinto et al. demonstrated the power of this paradigm in computer vision with Tuning Computer Vision Models With Task Rewards. Computer vision models are traditionally trained on standard cross-entropy or regression losses that suffer from the same token-level blindness found in NLP. By leveraging reinforcement learning to tune pretrained models directly on complex, non-differentiable downstream metrics like mean Average Precision (mAP) or Panoptic Quality (PQ), the authors achieved massive performance gains across object detection, segmentation, and image colorization.
By shifting our training objectives away from rigid data imitation and toward the direct optimization of task rewards, we allow models to learn the nuances, structural requirements, and semantic flexibilities that define real-world task success.
Key Foundational Work
Minimum Risk Annealing for Training Log-Linear Models (2006) — contributed 1 concept elements to this direction.
Minimum Error Rate Training in Statistical Machine Translation (2003) — contributed 1 concept elements to this direction.