In supervised learning, optimization is anchored by stable, deterministic targets. If a model is training to predict the next word in a sentence, the cross-entropy loss provides a direct, localized gradient for every single token.
Reinforcement learning (RL) enjoys no such luxury. When optimizing a policy via policy gradient methods, the training signal is derived from trial-and-error exploration. The standard policy gradient estimator relies on the product of the log-probability of an action and its associated return:
$$g = \mathbb{E}_{\tau \sim \pi} \left[ \sum_{t} \nabla_\theta \log \pi_\theta(a_t | s_t) G_t \right]$$
Because the return $G_t$ is an empirical sum of rewards collected over an entire trajectory, it introduces massive stochasticity. If an agent samples a sequence of actions that randomly encounters a high reward or a catastrophic penalty, the resulting gradient update can fluctuate by orders of magnitude. This inherent volatility—known as policy gradient variance—leads to frequent spikes in gradient norms, training instability, and catastrophic model collapse.
The challenge worsens when scaling RL to high-capacity sequence models like Large Language Models (LLMs). To understand why policy gradients fail in these regimes, and how modern techniques stabilize them, we must look at the root causes of variance and the mathematical mechanisms designed to tame it.
The Root Causes of Gradient Volatility
Gradient variance does not stem from a single source; rather, it is amplified by the environment's reward structure, the architecture of the model, and the mechanics of the optimization loop.
Unbounded and Noisy Reward Landscapes
When reward functions lack strict bounds, outlier trajectories can overwhelm the parameter updates. As Jiayi Fu et al. observe in the context of Reinforcement Learning from Human Feedback (RLHF), unconstrained reward models allow language models to "hack" the objective—maximizing scores through degenerate behaviors like extreme verbosity. These extreme reward values generate massive gradient updates that destabilize the value estimation components of the system.
Similarly, rare but catastrophic events introduce severe gradient spikes. Kamil Andrzej Ciosek et al. demonstrate that when an agent occasionally encounters a rare event with a massive penalty, the variance of the stochastic gradient explodes, causing the policy to diverge and lose previously stable behaviors.
The Critic Dilemma
To combat trajectory-level variance, actor-critic frameworks introduce an auxiliary "critic" network to estimate the expected value of a state, replacing the long-term return $G_t$ with a localized advantage estimate. However, this introduces a chicken-and-egg problem: the policy is optimized using gradients from the critic, but the critic itself is trying to learn an unstable, non-stationary value function.
Scott Fujimoto et al. show that updating a policy based on an un-converged or overestimating value function introduces high variance into the learning loop, creating a cycle where poor policy updates further complicate the critic's ability to provide accurate gradients. In autoregressive language models, this issue is magnified. Yiran Guo et al. point out that training a critic to estimate the value of partial sequences or individual tokens is highly unstable because the true value of a partial sentence varies wildly based on the initial prompt, making token-level value estimation an incredibly noisy signal.
Mechanism 1: Relative Baselines and Group Normalization
The most mathematically direct way to reduce variance without introducing bias is to subtract a baseline $B(s)$ from the return. Because $\mathbb{E}[\nabla_\theta \log \pi_\theta(a|s) B(s)] = 0$, we can alter the magnitude of the multiplier without shifting the expected value of the gradient:
$$g = \mathbb{E} \left[ \nabla_\theta \log \pi_\theta(a|s) (Q(s,a) - B(s)) \right]$$
The intuition is simple: we should not reward an action simply because it received a positive score; we should reward it only if it performed better than expected.
Self-Critical Baselines
In sequence-to-sequence tasks like image captioning, computing an accurate learned baseline for every generated token is computationally expensive and unstable. To bypass this, Steven J. Rennie et al. introduced Self-Critical Sequence Training (SCST).
Trajectory 1 (Sampled): "A dog catching a frisbee in flight" --> Reward: 0.85
Trajectory 2 (Greedy): "A dog catching a frisbee" --> Reward: 0.70 (Baseline)
-------------
Advantage: +0.15
Instead of training a separate critic network, SCST uses the reward obtained by the model's own inference-time greedy decoding as the baseline. The model acts as its own critic: if a stochastically sampled caption scores higher than its current deterministic "best guess," the gradient updates parameters to favor those sampled tokens. If it scores lower, the gradient acts as a negative push. This self-critical baseline normalizes the reward signal directly at the sequence level, dramatically stabilizing convergence.
Group Relative Optimization
While SCST compares a single sample against a greedy baseline, modern language model training often leverages batch-level distributions. Group Relative Policy Optimization (GRPO) eliminates the critic entirely by sampling a group of outputs $\{q_1, q_2, \dots, q_N\}$ for a single prompt and normalizing their rewards within the group.
Nghi D. Q. Bui et al. build upon this by introducing token-regulated adjustments to manage optimization sharpness. By weighting individual tokens based on their generation probabilities within the group, they suppress the sudden gradient norm spikes that typically plague complex reasoning tasks, ensuring smooth, monotonic policy improvements.
Mechanism 2: Reward Shaping and Landscape Smoothing
If the underlying variance is driven by an erratic or unbounded reward function, another powerful lever is to reshape the reward landscape itself to be more amenable to stochastic gradient descent.
Bounding via Sigmoidal Centering
To prevent extreme reward values from hijacking the policy gradient, Jiayi Fu et al. developed the Preference As Reward (PAR) framework. PAR applies a sigmoidal transformation to centered rewards:
$$R_{\text{shaped}} = \sigma(\alpha \cdot (R - R_0))$$
Unbounded Reward Space Bounded PAR Reward Space
+15.0 -------------------------> +1.0 (Asymptote)
/
+1.2 -------------------------> +0.6 / Linear-like region
-0.5 -------------------------> +0.4 \ near center
\
-12.0 -------------------------> +0.0 (Asymptote)
By passing the raw rewards through a squashing function, the authors strictly bound the feedback signal. Outlier trajectories that would normally generate massive, destabilizing gradient updates are compressed into the asymptotes of the sigmoid, while the model remains highly sensitive to subtle quality differences in the central, linear-like region of the function.
Distributional Smoothing
Standard policy gradient methods optimize for the expected return, which collapses a complex web of potential future outcomes into a single scalar. If an environment features a massive range of potential returns, this scalar expectation can shift violently from batch to batch.
Ke Sun et al. demonstrate that treating reinforcement learning from a distributional perspective—learning the full probability distribution of returns rather than just the mean—fundamentally smoothens the loss landscape. This distributional framing provides uniform stability during stochastic gradient descent, reducing the sample complexity required for stable convergence from $O(1/\tau^4)$ to $O(1/\tau^2)$.
Mechanism 3: Structural Credit Assignment and Decoupling
A final frontier in variance reduction involves changing where and how the gradient interacts with the model architecture.
Segment-Level Credit Assignment
When applying RL to long-form text generation or multi-step reasoning, developers face a stark trade-off: sequence-level rewards suffer from delayed feedback and high variance, while token-level rewards require an unstable, complex token critic.
Yiran Guo et al. resolve this tension with Segment Policy Optimization (SPO). Instead of evaluating the entire response or every individual token, SPO estimates rewards at the segment level (e.g., phrases or logical sentences).
Token-Level: [Token 1] -> [Token 2] -> [Token 3] -> [Token 4] (High Critic Instability)
Sequence-Level: [----------------- Full Response -----------------] (High Delayed Variance)
SPO (Segment): [ Segment 1 (Idea A) ] | [ Segment 2 (Idea B) ] (Goldilocks Zone)
By breaking the credit assignment down into meaningful semantic chunks, SPO achieves the precision of fine-grained optimization without the variance and architectural overhead of training an auxiliary token-level critic.
Architectural Decoupling
Directly propagating high-variance reinforcement learning gradients through massive neural network backbones can corrupt pre-trained representations. For instance, in visual RL, updating a heavy convolutional or transformer encoder with volatile policy gradients often causes the model to capture transient noise rather than robust features.
To protect high-capacity encoders from this volatility, Yunpeng Zhai et al. propose an asymmetric architecture. They decouple feature extraction from policy optimization by using a small, lightweight "adaptor" network to handle the sensitive, high-variance RL signals, while the large backbone focuses entirely on extracting stable visual features. This structural separation ensures that even if a gradient spike occurs, its impact is isolated to a small set of parameters, preventing a total training collapse.
Conclusion
The history of stabilizing policy gradients is a transition from raw trial-and-error to structured, bounded optimization. Whether by subtracting self-critical baselines (Rennie et al.), squashing rewards into stable bounds (Fu et al.), or re-architecting credit assignment (Guo et al.), the core objective remains identical: isolating the true signal of an action from the surrounding environmental noise. By understanding and mitigating these sources of variance, we can successfully deploy reinforcement learning to the world's most complex sequence and reasoning models.