Deep Reinforcement Learning and the Deadly Triad

Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, Joseph ModayilarXiv 2018 · OpenReview

Abstract

We know from reinforcement learning theory that temporal difference learning can fail in certain cases. Sutton and Barto (2018) identify a deadly triad of function approximation, bootstrapping, and off-policy learning. When these three properties are combined, learning can diverge with the value estimates becoming unbounded. However, several algorithms successfully combine these three properties, which indicates that there is at least a partial gap in our understanding. In this work, we investigate the impact of the deadly triad in practice, in the context of a family of popular deep reinforcement learning models - deep Q-networks trained with experience replay - analysing how the components of this system play a role in the emergence of the deadly triad, and in the agent's performance.

Introduction

In reinforcement learning (RL) theory, the deadly triad refers to the combination of function approximation, bootstrapping, and off-policy learning. Sutton and Barto (2018) proved that this combination can lead to unbounded value estimates, causing learning to fail. Despite this theoretical risk, modern deep RL successes like Deep Q-Networks (DQN) rely on all three components. This research bridges the gap between theory and practice, investigating why DQN remains stable and identifying the conditions under which the triad becomes dangerous. Through extensive empirical testing on 57 Atari games, the study finds that while "soft divergence" (unrealistic value spikes) is common, actual unbounded divergence is rare.

Methods and Experimental Setup

The authors conducted a massive empirical study using a standard DQN agent as a baseline. They systematically varied the "strength" of each triad component across the Arcade Learning Environment (ALE) suite of 57 Atari 2600 games:

  • Bootstrapping: Varied multi-step returns ($n=1, 3, 10$). Increasing $n$ reduces bootstrapping reliance.
  • Function Approximation: Tested four network sizes (small to extra-large) to evaluate how capacity affects stability.
  • Off-policy Strength: Manipulated the replay buffer using Prioritized Experience Replay; higher prioritization increases off-policy "strength."
  • Update Rules: Compared standard Q-learning, Target Q-learning, Double Q-learning, and "Inverse Double Q-learning" to isolate the effects of overestimation bias versus target networks.

The study involved three replications for each of the 336 parameter configurations per game, creating a comprehensive dataset of learning dynamics.

Key Insights: Soft Divergence and Self-Correction

A primary finding is the distinction between "hard" divergence (values hitting infinity) and "soft" divergence. In deep RL, hard divergence is nearly non-existent. Instead, value estimates often spike to unrealistic levels before potentially recovering.

The classic Tsitsiklis and Van Roy example (a) shows divergence (b), but adding parameters (u) can lead to convergence (c).

Surprisingly, the authors demonstrated that adding more learnable parameters can help an agent recover from initial divergence. As shown in the toy example above, while a simple model might diverge, a more complex one can "spiral out" and eventually find its way back to plausible value estimates.

Results: Stability and Performance

The empirical data highlights several critical factors for maintaining stability:

  • The Power of Double Q-Learning: Double Q-learning significantly improved stability, suggesting that the deadly triad is often worsened by simple overestimation bias.
  • Multi-step Returns: Increasing the bootstrap length to $n=10$ reduced the divergence rate from 94% to 21%.
  • The Capacity Paradox: Larger networks exhibited more frequent soft divergence but often achieved better final performance, challenging the idea that larger networks are inherently more stable.
Distribution of maximum absolute Q-values across different algorithms and network sizes.

As illustrated in the violin plots, standard Q-learning shows a high percentage of runs (61%) with unrealistic value estimates (above the dashed line at 100). In contrast, Double Q-learning reduces this to just 10%. There is a clear negative correlation between these unrealistic spikes and actual game performance; stability in $|Q|$ values is a strong predictor of higher scores.

Conclusion

While the "deadly triad" remains a theoretical threat, practical deep RL agents employ mechanisms that mitigate its effects. Target networks and Double Q-learning are essential stabilizers. Future research is needed to determine if these self-correcting dynamics are inherent to deep neural network loss landscapes or specific to common architectural choices like ReLU activations and the Adam optimizer.

Summary updated

Abstract

We know from reinforcement learning theory that temporal difference learning can fail in certain cases. Sutton and Barto (2018) identify a deadly triad of function approximation, bootstrapping, and off-policy learning. When these three properties are combined, learning can diverge with the value estimates becoming unbounded. However, several algorithms successfully combine these three properties, which indicates that there is at least a partial gap in our understanding. In this work, we investigate the impact of the deadly triad in practice, in the context of a family of popular deep reinforcement learning models - deep Q-networks trained with experience replay - analysing how the components of this system play a role in the emergence of the deadly triad, and in the agent's performance
BibTeX
@misc{hasselt2018deep,
  author = {Hado van Hasselt and Yotam Doron and Florian Strub and Matteo Hessel and Nicolas Sonnerat and Joseph Modayil},
  title = {{Deep Reinforcement Learning and the Deadly Triad}},
  year = {2018},
  eprint = {1812.02648},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.02648},
}
Figure