In supervised learning, we generally operate under a comfortable assumption: our training data is sampled from a fixed, stationary distribution. If we sample images of cats and dogs, the underlying features that define a cat do not change based on how well our model currently classifies them.
Deep reinforcement learning (RL) completely breaks this assumption. In RL, the data distribution is fundamentally coupled with the agent's behavior. As the agent learns, its policy updates; as its policy updates, it chooses different actions, thereby altering the distribution of states it encounters and the rewards it receives. This phenomenon is known as policy-induced non-stationarity.
Instead of optimizing over a stable landscape, an RL agent must constantly chase a moving target. This internal non-stationarity introduces severe optimization challenges that cause standard deep learning architectures to fail.
The Dual Nature of Non-Stationarity
To understand why non-stationarity destabilizes neural networks, we must look at how it manifests in two distinct components of the learning loop: the inputs (the data distribution) and the outputs (the optimization targets).
1. Shifting Input Distributions
As an agent's policy improves, the states it observes shift continuously. Early in training, an agent explores randomly, experiencing a highly volatile and noisy distribution of the environment. Later, as it converges on an optimal policy, it visits a much narrower, high-reward subspace.
As Huan Wang et al. note in On the Generalization Gap in Reparameterizable Reinforcement Learning, this continuous alteration of encountered states and actions makes it incredibly difficult to establish stable generalization guarantees. This early volatility can have long-lasting consequences. In Transient Non-stationarity and Generalisation in Deep Reinforcement Learning, Maximilian Igl et al. demonstrate that the noisy features acquired during early exploration can become deeply embedded in the network, making them difficult for the model to purge or replace even after the data distribution stabilizes.
2. Shifting Optimization Targets
In value-based RL, the network does not have access to ground-truth labels. Instead, it relies on bootstrapping—predicting the value of a state based on its own subsequent predictions (e.g., Temporal Difference learning).
This creates a severe "moving target" problem. When the network updates its parameters to match a target, it simultaneously changes the function that calculates that very target. Anas Barakat et al., in their formal analysis in Analysis of a Target-Based Actor-Critic Algorithm with Linear Function Approximation, show that updating a value function against an actively shifting target complicates the optimization landscape and can easily lead to divergent learning dynamics.
The Cost of Non-Stationarity: Plasticity Loss
When a standard deep neural network is forced to sequentially fit a constantly changing sequence of targets, it undergoes a pathological transformation: it loses its plasticity (or capacity)—the fundamental ability to learn new information.
[Evolving Policy] ──> [Shifting Data & Targets] ──> [Representational Decay]
▲ │
│ ▼
[Suboptimal Actions] <── [Dormant Neurons & Rigid Weights] ◄┘
In Understanding and Preventing Capacity Loss in Reinforcement Learning, Clare Lyle et al. show that this sequential fitting process causes internal representations to become saturated or trapped in low-dimensional subspaces that are no longer flexible enough to accommodate new functional mappings.
As the network struggles with this shifting landscape, its internal structure degrades. Clare Lyle et al. further demonstrate in Understanding Plasticity in Neural Networks that this loss of plasticity is driven by a progressive sharpening of the loss landscape and optimizer instability, rather than simple weight explosion.
This creates a destructive feedback loop. As explored by Hongyao Tang et al. in Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn, shifts in the data distribution lead to representational decay and increased prediction interference. This manifests empirically as a proliferation of "dormant neurons"—units that lose their capacity to activate or propagate meaningful gradients, effectively locking the network into a rigid, unadaptable state (Timon Willi et al. in Mixture of Experts in a Mixture of RL settings).
Why Standard Scaling Recipes Collapse
In computer vision or natural language processing, the standard recipe for better performance is simple: scale up the network. In deep RL, however, scaling up dense networks frequently causes performance to collapse.
Non-stationarity exacerbates standard gradient pathologies like vanishing signals. Traditional normalization layers (such as Batch Normalization) fail because they rely on running statistics that assume a stable data distribution. As Lu Li et al. explain in Normalization Enhances Generalization in Visual Reinforcement Learning, when an agent's experiences change rapidly, traditional batch statistics become highly unreliable, destroying the network's visual representations.
Furthermore, trying to dynamically normalize wildly varying reward scales introduces severe target non-stationarity. As Matteo Hessel demonstrates in Multi-Task Deep Reinforcement Learning with PopArt, constantly shifting the target values to accommodate new scales causes inconsistent updates that prevent value functions from converging stably.
Mitigating Non-Stationarity
To prevent networks from becoming rigid and unstable under policy-induced shifts, researchers have developed specialized architectural and algorithmic mechanisms designed to isolate representations and stabilize targets.
1. Modular and Sparse Architectures (Mixture of Experts)
Instead of forcing a single dense network to continuously overwrite its weights, researchers have turned to Mixture-of-Experts (MoE) modules. By routing different data distributions to distinct sparse pathways, MoEs prevent prediction interference and preserve plasticity.
Johan Obando-Ceron et al. show in Mixtures of Experts Unlock Parameter Scaling for Deep RL that replacing dense layers with Soft MoE modules successfully unlocks parameter scaling in value-based agents by insulating the model from non-stationary learning objectives. Similarly, Ghada Sokar et al. leverage Dynamic Sparse Training for Deep Reinforcement Learning to dynamically evolve a network's topology during training, allowing the architecture to adapt flexibly to changing data patterns without saturating its capacity.
2. Multi-Timescale Optimization
To mathematically tame the moving target problem, modern RL relies heavily on decoupling target generation from the active optimization loop. By utilizing a separate "target network" that updates on a slower, delayed timescale, the algorithm breaks the immediate feedback loop between a policy update and its regression target. Anas Barakat et al. provide formal proofs demonstrating that this multi-timescale framework provides the necessary stability to guarantee convergence despite the underlying non-stationarity.
3. Feature Regularization
Rather than allowing the network to freely collapse its representations into low-dimensional subspaces, explicit regularization can be used to preserve capacity. In Understanding and Preventing Capacity Loss in Reinforcement Learning, Clare Lyle et al. introduce Initial Feature Regularization (InFeR). By penalizing the network when its current features drift too far from their initial random projections, they anchor the representational space, ensuring the network retains the structural flexibility needed to master new targets late into training.