When training deep neural networks on a single, static dataset, gradient descent works remarkably well. But when a network faces a non-stationary environment—such as a sequence of different tasks or a reinforcement learning (RL) environment with shifting reward signals—a strange pathology emerges. Over time, the network becomes "stiff" and increasingly resistant to new information.
This phenomenon is known as the Loss of Plasticity (LoP) or Capacity Loss. As Wang et al. observe in The Dual Nature of Plasticity Loss in Deep Continual Learning: Dissection and Mitigation, models experience a progressive decline in their capacity to acquire new information after training on long sequences of tasks. Crucially, this occurs even when the network has ample parameter capacity.
To build intuition, it helps to contrast plasticity loss with its famous cousin, catastrophic forgetting. Forgetting is a backward-looking failure: the network learns Task B but forgets Task A. Plasticity loss is a forward-looking failure: the network has learned Task A so rigidly that it cannot even begin to learn Task B. In fact, a highly trained network suffering from plasticity loss will often perform worse at learning a new task than a completely random, freshly initialized network.
Why do deep networks lose their ability to learn, and how can we keep them adaptable?
The Causal Mechanisms of Rigidity
Plasticity loss is not caused by a single bug; rather, it is an emergent property of standard deep learning components interacting with non-stationary data. Researchers have isolated several distinct structural and geometric failures that drive this behavior.
1. The Accumulation of Dormant Neurons
In networks utilizing Rectified Linear Units (ReLUs), a neuron outputs zero if its input is negative. If a neuron's weights are updated such that it outputs negative values across the entire data distribution, its gradient becomes exactly zero. It dies.
In static supervised learning, dead ReLUs happen but are manageable. In non-stationary environments, however, they become a cumulative disease. Sokar demonstrates in The Dormant Neuron Phenomenon in Deep Reinforcement Learning that as target values continuously shift, neurons lose relevance to the evolving objective and permanently turn off. Because standard gradient descent cannot propagate updates through a dead ReLU, these units never recover on their own. This permanently shrinks the functional capacity of the network over time.
2. Representation and Rank Collapse
Even if neurons remain technically active, the geometric structure of the network's representations can collapse. Lyle et al. show in Understanding and Preventing Capacity Loss in Reinforcement Learning that internal representations tend to shrink into a lower-dimensional subspace.
Mathematically, this manifests as a rank collapse of the Neural Tangent Kernel (NTK). As Tang et al. argue in Mitigating Plasticity Loss in Continual Reinforcement Learning by Reducing Churn, a drop in NTK rank means that the gradients for different data points become highly correlated. Geometrically, this forces all weight updates into a narrow, restrictive subspace. The network loses the expressivity required to distinguish between new environmental states or to map novel inputs to distinct outputs.
[High-Rank Representation] [Rank Collapse]
Inputs map widely into Inputs compressed into
a rich 3D state space a rigid 1D line
(Flexible) (Unadaptable)
3. Loss Landscape Sharpening
Optimization algorithms are inherently biased toward certain regions of the parameter space. Lyle et al. demonstrate in Understanding Plasticity in Neural Networks that gradient descent naturally drives network parameters toward sharper regions of the loss landscape.
In these sharp valleys, gradients from different data points interfere with one another aggressively. An update that helps satisfy a new objective completely destroys the network's stability elsewhere, causing optimization to stall. This structural change in the parameter space reduces the network's ongoing trainability, independent of any capacity limits.
4. Weight Pathologies
Over long training horizons, parameter norms can explode or degenerate. For instance, Zilly identifies a phenomenon called "mutually frozen weights" in On Plasticity, Invariance, and Mutually Frozen Weights in Sequential Task Learning, where weight decay causes parameters in successive layers to zero each other out. This effectively kills gradient propagation and locks the network's state.
Strategies for Maintaining Plasticity
To prevent networks from freezing, researchers have developed interventions that target these mechanisms, categorized into resets, architectural changes, and regularization.
Neuron Recycling and Resets
If the root cause is that parameters become stuck or dead, the most direct solution is to reset them.
- ReDo (Recycling Dormant Neurons): Sokar proposes periodically identifying dormant neurons during training and re-initializing them to fresh, random values, maintaining network expressivity over long horizons.
- Self-Normalized Resets (SNR): Farias et al. expand on this in Self-Normalized Resets for Plasticity in Continual Learning by introducing a statistically grounded detection mechanism that resets neurons based on their individual firing histories, preventing the network from becoming "stiff."
Architectural Interventions
Instead of modifying weights post-hoc, we can change the network's structure to inherently resist collapse.
- Mixture of Experts (MoE): Obando-Ceron et al. demonstrate in Mixtures of Experts Unlock Parameter Scaling for Deep RL that replacing dense layers with Soft MoE modules prevents layers from becoming dormant. This distributes the learning burden and unlocks parameter scaling in value-based RL.
- Alternative Activations: Lewandowski introduces Plastic Learning with Deep Fourier Features, replacing standard activations with sine and cosine pairs. This maintains a dynamic balance between the trainability of linear models and the expressive power of deep networks, bypassing ReLU death entirely.
Landscape Smoothing and Regularization
We can also alter the optimization trajectory to avoid sharp regions and maintain gradient flow. Cho introduces the PLASTIC algorithm in PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learning, which combines loss-landscape smoothing via Sharpness-Aware Minimization (SAM) and Layer Normalization with Concatenated ReLUs (CReLU). This multi-pronged defense ensures smooth gradient propagation and prevents representational collapse even under high-intensity data reuse.