Self-Critical Sequence Training for Metric Optimization

306 concepts223 papers3 proposals

In sequence generation tasks—such as image captioning, summarization, and machine translation—models are traditionally trained using Maximum Likelihood Estimation (MLE). This approach minimizes the cross-entropy loss by predicting the next token given a ground-truth prefix. However, this creates a fundamental mismatch: models are trained to maximize word-level probability but are evaluated on sequence-level metrics like CIDEr, BLEU, or ROUGE.

To bridge this gap, researchers have turned to reinforcement learning (RL) to optimize these non-differentiable metrics directly. The most prominent framework for this is Self-Critical Sequence Training (SCST).

The Problem of Exposure Bias

Standard MLE training suffers from "exposure bias," a phenomenon where a model only sees ground-truth data during training but must rely on its own potentially erroneous predictions during inference. As Sakriani Sakti et al. note in Sequence-to-Sequence ASR Optimization via Reinforcement Learning, this leads to error accumulation over time. By treating sequence generation as a reinforcement learning problem, the model can explore its own predicted trajectories and learn to recover from mistakes, effectively aligning the training environment with the inference-time reality.

The Self-Critical Mechanism

The core innovation of SCST is how it defines the "reward" for the RL agent. In a typical policy gradient setup, the model samples a sequence and receives a reward based on how well that sequence scores on a target metric. However, these rewards can be highly volatile, making training unstable.

SCST stabilizes this by using the model's own current performance as a baseline. As explained by Luke Melas-Kyriazi et al. in Training for Diversity in Image Paragraph Captioning, the model generates two sequences:

  1. A sampled sequence drawn from the current probability distribution.
  2. A baseline sequence generated via greedy search (picking the most likely word at each step).

The model is rewarded only if the sampled sequence outperforms its own greedy baseline. This relative reward structure ensures that the model is constantly pushed to improve upon its current best guess without requiring external reference scores during the RL phase, as highlighted by Clément Rebuffel et al. in PARENTing via Model-Agnostic Reinforcement Learning to Correct Pathological Behaviors in Data-to-Text Generation.

Beyond Simple Word Matching

One of the greatest strengths of metric-optimized RL is its flexibility. Because the reward function does not need to be differentiable, researchers can optimize for complex, human-centric qualities that cross-entropy loss cannot capture:

Broad Impact Across Domains

While initially popularized in image captioning—where it is used to optimize the CIDEr metric (Simao Herdade et al., Image Captioning: Transforming Objects into Words)—this approach has spread across the AI landscape. It has been successfully applied to video description (Bairui Wang et al., Reconstruct and Represent Video Contents for Captioning via Reinforcement Learning), surgical instruction generation (Jinglu Zhang et al., Surgical Instruction Generation with Transformers), and even logistics route prediction (Haomin Wen et al., DRL4Route: A Deep Reinforcement Learning Framework for Pick-up and Delivery Route Prediction).

By shifting the training objective from local token accuracy to global sequence quality, self-critical reinforcement learning ensures that generative models don't just "play it safe" with generic outputs, but instead produce text that is coherent, descriptive, and aligned with human evaluation.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 232 papers
  2. Version 1 · 2026-10-04 · 333 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_self_critical_sequence_training_d80ba49c,
  title = {self-critical sequence training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/self-critical-sequence-training/rdr_d80ba49ce50c47059be0285e48afcb0b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_self_critical_sequence_training_d80ba49c_v2,
  title = {self-critical sequence training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/self-critical-sequence-training/rdr_d80ba49ce50c47059be0285e48afcb0b/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake