Mitigating Policy Oscillation and Collapse in Reinforcement Learning

50 concepts49 papers2 proposals

In supervised learning, if a model makes a poor optimization update, the underlying dataset remains unchanged. In reinforcement learning (RL), however, the agent's policy directly dictates its future data stream. This coupling creates a fragile feedback loop: a single suboptimal update can cause the agent to collect low-quality or erratic experience, which in turn poisons subsequent gradient steps.

This fundamental vulnerability manifests in two primary ways: policy oscillation (or chattering), where the agent's strategy fluctuates endlessly between suboptimal behaviors, and policy collapse, where performance abruptly and catastrophically plummets. As Paul Wagner demonstrates in A reinterpretation of the policy oscillation phenomenon in approximate policy iteration, fast "greedy" value-based methods often act as extreme, overshooting versions of policy gradients, leading to severe numerical and behavioral instability.

The Geometry of Collapse: Why Agents Dive Off Cliffs

To understand why updates fail catastrophically, we must look at the underlying optimization landscape. In Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environments, Ryan Sullivan et al. mapped the reward surfaces of popular benchmarks, revealing that they are rarely smooth. Instead, they contain literal "cliffs"—sharp, discrete drops in performance where a minor parameter shift plunges the agent into low-reward or absorbing failure states.

When an unconstrained update pushes the policy over one of these edges, the data-collection mechanism breaks. As Maximilian Igl notes in Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneck, standard stochastic regularization or exploration noise can inadvertently trigger this collapse. If the noise causes erratic actions, the resulting low-quality experience forms a destructive feedback loop that completely halts the discovery of optimal behaviors. This sensitivity is closely tied to the mathematical "health" of the policy; Arip Asadulaev et al. in Exploring and Exploiting Conditioning of Reinforcement Learning Agents show that poor Jacobian conditioning of the policy network serves as a strong predictor for these sudden performance drops.

Breaking the Loop via Conservative Updates and Anchoring

To prevent agents from overshooting or diving off reward cliffs, researchers have developed mechanisms that enforce cautious, incremental policy shifts. The core intuition is simple: instead of immediately transitioning to a fully greedy policy based on noisy, approximate value estimates, updates must be regularized, bounded, or averaged.

In Deep Conservative Policy Iteration, Nino Vieillard scales classic conservative policy bounds to deep neural networks, demonstrating that smoothing out the transition to greedy policies mitigates the extreme volatility caused by function approximation errors. Similarly, Alberto Maria Metelli et al. in Safe Policy Iteration: A Monotonically Improving Approximate Policy Iteration Approach formalize this by maximizing a theoretically derived lower bound on policy improvement, ensuring that the agent's performance improves monotonically without destructive oscillations.

Another powerful approach to stabilization is variance reduction through temporal averaging or anchoring. For value-based methods, Oron Anschel et al. introduce Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning, which averages Q-value estimates across previous iterations to dampen erratic updates. For policy-based methods, Alena Shilova et al. develop StaQ it! Growing neural networks for Policy Mirror Descent, utilizing a finite stack of historical Q-functions and a weight-correction scheme to approximate the theoretical stability of Policy Mirror Descent without requiring infinite memory.

The Modern Frontier: Policy Collapse in LLM Reasoning

The problem of policy collapse has re-emerged with severe intensity in the context of training Large Language Models (LLMs) to reason using reinforcement learning (e.g., via algorithms like GRPO or PPO). When language models are optimized using outcome-only or sparse rewards, the massive token action space generates highly volatile gradients.

As Hongming Wu et al. explain in M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for Large Language Models with Momentum-Anchored Policy Optimization, self-supervised LLM training often suffers from a catastrophic feedback loop where the model becomes overconfident in incorrect reasoning paths, leading to sudden performance crashes.

To stabilize these models, modern techniques target the specific tokens and trajectories driving the instability:

Key Foundational Work

A reinterpretation of the policy oscillation phenomenon in approximate policy iteration (2011) — contributed 1 concept elements to this direction.
Reinforcement Learning Via Practice and Critique Advice (2010) — contributed 1 concept elements to this direction.
Relative Entropy Policy Search (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_training_instability_93ab9f35,
  title = {training instability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/training-instability/rdr_93ab9f3564af4e84b8df255a644f5997},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_training_instability_93ab9f35_v1,
  title = {training instability},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/training-instability/rdr_93ab9f3564af4e84b8df255a644f5997/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake