Self-Critical Reinforcement Learning for Sequence Generation

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2333 papers386 conceptsfrozen 2026-10-04

Most sequence-to-sequence models are trained using Maximum Likelihood Estimation (MLE), where the model learns to predict the next word given the ground-truth prefix. While mathematically convenient, this approach suffers from two fundamental flaws: exposure bias and metric mismatch.

Exposure bias occurs because the model is only "exposed" to perfect human-written prefixes during training, but must rely on its own potentially erroneous predictions during inference. Metric mismatch arises because models are trained to minimize cross-entropy loss (a word-level probability) but are evaluated on sequence-level metrics like BLEU, ROUGE, or CIDEr.

Self-critical reinforcement learning addresses these issues by allowing the model to "practice" its own generation and receive rewards based on the final quality of the entire sequence.

The Self-Critical Baseline

The core challenge in applying reinforcement learning to text is the high variance of the policy gradient. If a model generates a sentence and receives a score, how does it know if that score is "good" or "bad" relative to its current capability?

The breakthrough, popularized as Self-Critical Sequence Training (SCST), is to use the model's own current performance as a baseline. As Junlong Gao and Shiqi Wang describe in Self-Critical N-Step Training for Image Captioning, the model generates two sequences:

  1. A sampled sequence where words are chosen stochastically.
  2. A greedy sequence where the model always picks the most likely word (its current "best guess").

The reward for the sampled sequence is compared against the reward of the greedy sequence. If the sample outperforms the greedy baseline, the model's parameters are updated to make that sample more likely. If it performs worse, the probability is decreased. This "self-critical" mechanism ensures that the model only reinforces sequences that exceed its own average capability, significantly stabilizing training.

Bridging Training and Evaluation

By treating sequence generation as a series of actions in a Reinforcement Learning (RL) framework, researchers can optimize for non-differentiable metrics that more closely align with human judgment.

In the domain of image and video captioning, Boqing Gong and Lijun Li demonstrated in End-to-End Video Captioning with Multitask Reinforcement Learning that optimizing directly for CIDEr—a metric designed to capture human-like consensus—leads to more descriptive and accurate outputs than standard cross-entropy. Similarly, Caiming Xiong and Richard Socher applied this to speech recognition in Improving End-to-End Speech Recognition with Policy Learning, directly optimizing for Word Error Rate (WER) to bridge the gap between training objectives and test-time performance.

Beyond Simple Overlap Metrics

While early work focused on word-overlap metrics like BLEU, the framework has expanded to optimize for complex, qualitative attributes:

Stabilizing Long-Horizon Reasoning

As models tackle more complex tasks like autonomous search and multi-step tool use, the feedback signal becomes sparser. Bruce Yang and Huan Gao introduced DSPO in DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning, which calculates importance ratios across entire trajectories. This sequence-level alignment prevents the "policy collapse" that often occurs when a model receives a single reward at the end of a very long chain of thoughts.

By shifting the unit of optimization from the individual token to the entire sequence, self-critical reinforcement learning allows models to move past rote imitation and toward a more robust, goal-oriented generation strategy.

Papers in this version

Showing 60 of 333 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 232 papers
  2. Version 1 · 2026-10-04 · 333 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_self_critical_sequence_training_d80ba49c,
  title = {self-critical sequence training},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/self-critical-sequence-training/rdr_d80ba49ce50c47059be0285e48afcb0b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_self_critical_reinforcement_learning_for_d80ba49c_v1,
  title = {Self-Critical Reinforcement Learning for Sequence Generation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/self-critical-reinforcement-learning-for-sequence-generation/rdr_d80ba49ce50c47059be0285e48afcb0b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}