Direct Optimization of Evaluation Metrics for Generative Models

This direction is split into other directions: objective-metric misalignment (training-evaluation gap). Its last version stays readable and citable here.

Version 1 of 135 papers37 conceptsfrozen 2026-10-04

Most machine learning models are trained using a proxy. In natural language processing and computer vision, we typically train models to minimize Cross-Entropy Loss—essentially asking the model to predict the next token or pixel as accurately as possible.

However, we don't judge these models by their cross-entropy. We judge them by metrics like BLEU (for translation), CIDEr (for image captioning), or mAP (for object detection). This creates a fundamental "training-evaluation gap": a model can become statistically more probable without becoming more useful or accurate for its intended task.

The Misalignment of Likelihood

The core problem, as identified early on by Franz Josef Och in Minimum Error Rate Training in Statistical Machine Translation, is that Maximum Likelihood Estimation (MLE) is a local objective. It prioritizes matching the ground-truth data distribution token-by-token.

This leads to several issues:

  1. Exposure Bias: During training, the model only sees "perfect" prefixes from the dataset. At test time, it must generate from its own (potentially imperfect) previous outputs.
  2. Metric Insensitivity: MLE treats all "wrong" tokens equally. In a translation task, replacing a word with a synonym is penalized as heavily as replacing it with a random word, even though the former might still result in a high BLEU score.
  3. Global vs. Local: As noted in Stack-Captioning: Coarse-to-Fine Learning for Image Captioning, optimizing for individual word probability does not necessarily capture the overall meaning or descriptive accuracy of a sentence.

Bridging the Gap with Minimum Risk Training

To solve this, researchers moved toward Minimum Risk Training (MRT). Instead of maximizing the probability of the one correct answer, MRT minimizes the "expected loss" across a range of possible outputs.

In Minimum Risk Training for Neural Machine Translation, Maosong Sun and Yong Cheng demonstrated that by sampling multiple candidate translations and weighting them by their actual BLEU scores, the model learns to prioritize sequences that "feel" right to the evaluation metric. This approach was later extended to handle document-level context by Danielle Saunders and Bill Byrne in Using Context in Neural Machine Translation Training Objectives, ensuring that translations remain consistent across an entire text rather than just within a single sentence.

Reward-Augmented Learning

A more generalized framework for this is Reward Augmented Maximum Likelihood (RAML). Introduced by Mohammad Norouzi and colleagues in Reward Augmented Maximum Likelihood for Neural Structured Prediction, RAML acts as a bridge between supervised learning and reinforcement learning.

Instead of just training on the ground truth, RAML samples "nearby" outputs and assigns them rewards based on their similarity to the target (using metrics like edit distance). This smooths the optimization landscape. As Xuezhe Ma and Graham Neubig later interpreted in Softmax Q-Distribution Estimation for Structured Prediction, this effectively trains the model to match a distribution where high-reward outputs are exponentially more likely.

Beyond Text: Vision and Scientific Discovery

This philosophy of "training on the metric" has spread beyond translation:

  • Computer Vision: In Tuning Computer Vision Models With Task Rewards, André Susano Pinto and Alexander Kolesnikov showed that object detection and segmentation models improve significantly when tuned using Reinforcement Learning to maximize task-specific rewards like mAP.
  • Protein Design: The FIDIA framework by Minghan Li addresses a specific misalignment in scientific discovery: models are often trained for average performance, but in the lab, we only care about the single best candidate in a batch. FIDIA optimizes for this "Best-of-N" outcome, prioritizing the rare, high-quality sequences needed for vaccines and enzymes.

Efficiency and Stability

Directly optimizing metrics can be computationally expensive because it often requires generating many samples during training. Recent work has focused on making this more efficient. Takumi Takada and colleagues introduced Direct Metric Optimization for Image Captioning through Reward-Weighted Augmented Data Utilization, which uses offline data augmentation and reward-weighting to achieve RL-level performance at a fraction of the cost.

Similarly, Daniel Kang and Tatsunori Hashimoto proposed Improved Natural Language Generation via Loss Truncation, a method that adaptively drops high-loss examples. This prevents the model from wasting capacity trying to mimic noisy or hallucinated data, ensuring the training signal remains aligned with factual accuracy.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_direct_optimization_of_evaluation_metric_dba00e1a,
  title = {Direct Optimization of Evaluation Metrics for Generative Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/direct-optimization-of-evaluation-metrics-for-generative-models/rdr_dba00e1a36f04b72b7176452a7ced3ea},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_direct_optimization_of_evaluation_metric_dba00e1a_v1,
  title = {Direct Optimization of Evaluation Metrics for Generative Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/direct-optimization-of-evaluation-metrics-for-generative-models/rdr_dba00e1a36f04b72b7176452a7ced3ea/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}