TL;DR
This paper investigates the "edge of stability" phenomenon, previously observed in supervised learning, within the context of reinforcement learning (RL). The authors find that this phenomenon, where training dynamics become unstable near a critical point, can occur in off-policy RL algorithms like DQN, but its presence depends on the specific loss function used, highlighting key differences between RL and supervised learning optimization.
What Problem Does This Paper Address?
In supervised learning, researchers have identified an "edge of stability" phenomenon. This occurs when the training process, specifically the leading eigenvalue of the Hessian matrix, approaches a critical threshold. Beyond this threshold, training can become unstable in the short term, exhibiting oscillations, even though the overall loss might still decrease over longer periods. This phenomenon is crucial for understanding why neural networks train the way they do. However, it's unclear if this phenomenon translates to reinforcement learning, which has unique challenges like non-stationary data distributions and bootstrapping. This paper aims to bridge this gap by empirically investigating the edge of stability in off-policy deep RL algorithms.
Methods
The researchers explored the edge of stability phenomenon using two popular off-policy deep RL algorithms: DQN (Deep Q-Networks) and C51 (Categorical DQN). They employed gradient descent with and without momentum. To isolate the effects of RL-specific challenges, they experimented across various data regimes:
- Offline RL: Training on a fixed replay buffer generated by a pre-trained agent. This setup mimics supervised learning more closely.
- Online RL: Training where the agent's own experience populates the replay buffer, introducing non-stationarity.
They analyzed the leading eigenvalue of the Hessian and the loss function's behavior during training.
Datasets
The experiments were conducted on the MinAtar environment, a simplified version of Atari games designed for faster experimentation and to reduce visual complexity. The paper mentions that results from MinAtar have been shown to translate to full Atari games. Specific games used include Breakout and Space Invaders. The authors also state that they will make their code and data publicly available upon acceptance.
Key Insights and Ideas
- Edge of Stability in RL is Possible, but Conditional: The edge of stability phenomenon is not a universal feature of deep RL but can be induced, particularly in off-policy settings.
- Loss Function Matters: The choice of loss function significantly impacts whether the edge of stability is observed. DQN with a Huber loss exhibited the phenomenon, while C51 with a cross-entropy loss did not show a consistent effect. This is a key differentiator from supervised learning.
- RL's Unique Challenges: While neural network architectures might allow for some transfer of optimization dynamics across domains, RL's inherent complexities (non-stationarity, bootstrapping) can lead to distinct optimization behaviors compared to supervised learning.
- Offline RL as a Proxy: Offline RL settings, where data is fixed, provide a closer analog to supervised learning and are more likely to exhibit the edge of stability. Online RL, with its dynamic data, presents a more complex picture.
- Instability vs. Performance: The paper notes that while loss instability can occur at the edge of stability, it doesn't always directly translate to a degradation in the agent's reward.
Key Results
- DQN: In offline RL settings (using greedy, ε-greedy, or pre-trained agent replay buffers), DQN consistently showed a clear edge of stability effect. The leading eigenvalue of the Hessian rose to the quadratic divergence threshold, fluctuated around it, and the loss exhibited short-term instabilities. In online RL, this effect was diminished, with eigenvalues plateauing below the threshold.
- C51: C51 did not consistently exhibit the edge of stability phenomenon across all settings. In offline RL, the leading eigenvalue often stayed below the threshold. In online RL, eigenvalues significantly exceeded the threshold. However, C51 did show a correlation between large leading Hessian eigenvalues and training challenges.
- Full-batch vs. Mini-batch: Full-batch experiments generally reinforced the findings from mini-batch experiments, showing a strong edge of stability effect for DQN in offline settings.
Limitations and Future Work
- Environment Specificity: The experiments were conducted on the MinAtar environment. The authors hope future work will expand these findings to a wider range of environments.
- Optimizer Exploration: The study focused on gradient descent with momentum. Future research could explore adaptive optimizers like Adam, which are commonly used in RL.
- Connecting Eigenvalues to Performance: A key future direction is to connect the leading eigenvalue of the Hessian not just to the loss function's behavior but also to the agent's actual performance and generalization, similar to work done in supervised learning.
Summary updated