Most sequence-to-sequence models are trained using Maximum Likelihood Estimation (MLE), where the model learns to predict the next word given the ground-truth prefix. While mathematically convenient, this approach suffers from two fundamental flaws: exposure bias and metric mismatch.
Exposure bias occurs because the model is only "exposed" to perfect human-written prefixes during training, but must rely on its own potentially erroneous predictions during inference. Metric mismatch arises because models are trained to minimize cross-entropy loss (a word-level probability) but are evaluated on sequence-level metrics like BLEU, ROUGE, or CIDEr.
Self-critical reinforcement learning addresses these issues by allowing the model to "practice" its own generation and receive rewards based on the final quality of the entire sequence.
The Self-Critical Baseline
The core challenge in applying reinforcement learning to text is the high variance of the policy gradient. If a model generates a sentence and receives a score, how does it know if that score is "good" or "bad" relative to its current capability?
The breakthrough, popularized as Self-Critical Sequence Training (SCST), is to use the model's own current performance as a baseline. As Junlong Gao and Shiqi Wang describe in Self-Critical N-Step Training for Image Captioning, the model generates two sequences:
- A sampled sequence where words are chosen stochastically.
- A greedy sequence where the model always picks the most likely word (its current "best guess").
The reward for the sampled sequence is compared against the reward of the greedy sequence. If the sample outperforms the greedy baseline, the model's parameters are updated to make that sample more likely. If it performs worse, the probability is decreased. This "self-critical" mechanism ensures that the model only reinforces sequences that exceed its own average capability, significantly stabilizing training.
Bridging Training and Evaluation
By treating sequence generation as a series of actions in a Reinforcement Learning (RL) framework, researchers can optimize for non-differentiable metrics that more closely align with human judgment.
In the domain of image and video captioning, Boqing Gong and Lijun Li demonstrated in End-to-End Video Captioning with Multitask Reinforcement Learning that optimizing directly for CIDEr—a metric designed to capture human-like consensus—leads to more descriptive and accurate outputs than standard cross-entropy. Similarly, Caiming Xiong and Richard Socher applied this to speech recognition in Improving End-to-End Speech Recognition with Policy Learning, directly optimizing for Word Error Rate (WER) to bridge the gap between training objectives and test-time performance.
Beyond Simple Overlap Metrics
While early work focused on word-overlap metrics like BLEU, the framework has expanded to optimize for complex, qualitative attributes:
- Diversity and Specificity: In Training for Diversity in Image Paragraph Captioning, Luke Melas-Kyriazi and Alexander M Rush showed that SCST can be combined with repetition penalties to nearly double the quality of long-form descriptions.
- Coherence: For document-level translation, Hao Xiong and colleagues used RL to optimize for global discourse properties, ensuring that sentences flow logically rather than just being individually accurate Modeling Coherence for Discourse Neural Machine Translation.
- Subjective Qualities: Recent work like Reinforced Sequence Training based Subjective Bias Correction uses RL to strip subjective bias from text, optimizing for neutrality and fluency simultaneously.
Stabilizing Long-Horizon Reasoning
As models tackle more complex tasks like autonomous search and multi-step tool use, the feedback signal becomes sparser. Bruce Yang and Huan Gao introduced DSPO in DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning, which calculates importance ratios across entire trajectories. This sequence-level alignment prevents the "policy collapse" that often occurs when a model receives a single reward at the end of a very long chain of thoughts.
By shifting the unit of optimization from the individual token to the entire sequence, self-critical reinforcement learning allows models to move past rote imitation and toward a more robust, goal-oriented generation strategy.