Abstract
The reinforcement learning (RL) problem is rife with sources of non-stationarity, making it a notoriously difficult problem domain for the application of neural networks. We identify a mechanism by which non-stationary prediction targets can prevent learning progress in deep RL agents: capacity loss, whereby networks trained on a sequence of target values lose their ability to quickly update their predictions over time. We demonstrate that capacity loss occurs in a range of RL agents and environments, and is particularly damaging to performance in sparse-reward tasks. We then present a simple regularizer, Initial Feature Regularization (InFeR), that mitigates this phenomenon by regressing a subspace of features towards its value at initialization, leading to significant performance improvements in sparse-reward environments such as Montezuma's Revenge. We conclude that preventing capacity loss is crucial to enable agents to maximally benefit from the learning signals they obtain throughout the entire training trajectory.
Introduction
In supervised learning, models typically benefit from a fixed dataset and stationary targets. Reinforcement Learning (RL) presents a starkly different challenge: the "ground truth" is a moving target. As an agent explores its environment and updates its policy, the value functions it attempts to approximate change constantly. This inherent non-stationarity can lead to a phenomenon known as capacity loss, where a neural network becomes increasingly rigid and loses its "plasticity"—the ability to adapt to new information.
This paper identifies capacity loss as a primary reason why RL agents often fail in sparse-reward environments. Even if an agent eventually discovers a rare reward, its network may have already "hardened" or overfitted to the early, reward-less stages of training, rendering it unable to incorporate the new signal effectively.
Measuring Capacity and Representation Collapse
To quantify this phenomenon, the researchers utilize two primary metrics:
- Target-fitting capacity: This measures how effectively a network checkpoint can fit a brand-new, random target function within a limited number of gradient steps. A decrease in this ability over training time indicates capacity loss.
- Feature Rank: This serves as a mathematical proxy for capacity by measuring the dimensionality of the feature space (the penultimate layer). A low rank suggests "representation collapse," where the network maps many distinct states to the same internal representation, losing the nuance required for complex decision-making.
Initial Feature Regularization (InFeR)
To combat capacity loss, the authors introduce Initial Feature Regularization (InFeR). The core intuition is that at initialization, a network possesses a diverse, albeit random, set of features. InFeR seeks to preserve a portion of this initial diversity throughout the training process.
The Mechanism: InFeR attaches auxiliary "heads" to the network backbone. During training, these heads are subjected to an $L_2$ penalty that forces them to output the same values they would have produced at initialization. By regressing a subspace of features toward their initial state, InFeR prevents the backbone from collapsing into a low-dimensional state, effectively anchoring the network's plasticity.
Experimental Results
The authors evaluated InFeR across several benchmarks, including a non-stationary version of MNIST and the Atari 2600 suite.
Breakthroughs in Sparse-Reward Tasks
The most significant impact was observed in "hard exploration" games like Montezuma’s Revenge. Standard agents like Rainbow or DQN frequently score 0 on this task because they lose capacity before encountering rewards. By implementing InFeR, these same agents achieved scores of approximately 2500. This suggests that the bottleneck in these games is not just exploration, but the agent's ability to learn from rewards once they are finally found.
Feature Rank and Generalization
InFeR successfully maintained higher feature ranks across the Atari suite compared to baseline agents. While the most dramatic improvements occurred in sparse-reward settings, InFeR also improved average human-normalized scores across the entire 57-game benchmark, demonstrating its broad utility in maintaining model health.
Discussion and Limitations
A key insight of this work is that over-parameterization acts as a natural buffer against capacity loss; larger networks tend to retain plasticity longer than smaller ones. However, InFeR provides a targeted way to maintain this capacity regardless of model size.
There are trade-offs to consider. In environments with very "dense" and immediate rewards (e.g., Jamesbond), InFeR can occasionally slow down early learning. This suggests the regularization might be slightly too restrictive when the primary task already provides a sufficient signal to maintain representation diversity. Future work may explore more dynamic "anchor" representations that provide the benefits of InFeR without interfering with high-signal learning phases.
Conclusion
Preventing capacity loss is essential for deep RL agents to maximize the utility of the learning signals they encounter. By maintaining feature diversity through InFeR, agents can remain adaptable throughout their entire training trajectory, turning potential dead ends into successful learning opportunities.
Summary updated