Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear

Zachary C. Lipton, Jianfeng Gao, Lihong Li, Jianshu Chen, Li DengarXiv 2016 · OpenReview

Abstract

Many practical environments contain catastrophic states that an optimal agent would visit infrequently or never. Even on toy problems, Deep Reinforcement Learning (DRL) agents tend to periodically revisit these states upon forgetting their existence under a new policy. We introduce intrinsic fear (IF), a learned reward shaping that guards DRL agents against periodic catastrophes. IF agents possess a fear model trained to predict the probability of imminent catastrophe. This score is then used to penalize the Q-learning objective. Our theoretical analysis bounds the reduction in average return due to learning on the perturbed objective. We also prove robustness to classification errors. As a bonus, IF models tend to learn faster, owing to reward shaping. Experiments demonstrate that intrinsic-fear DQNs solve otherwise pathological environments and improve on several Atari games.

Introduction: The Sisyphean Curse

In Deep Reinforcement Learning (DRL), success can paradoxically lead to failure. When an agent successfully learns to avoid a catastrophic state—such as a robot falling off a ledge or a vehicle crashing—it stops experiencing that state. In standard Deep Q-Networks (DQNs), experiences are typically stored in a sliding-window "experience replay buffer." As the agent's policy improves and it becomes safer, data regarding these catastrophes is eventually evicted from the buffer to make room for new, safe transitions.

Consequently, the neural network's weights are updated using only "safe" data. This leads to catastrophic forgetting: the Q-values for dangerous states drift, and the agent eventually "forgets" why it was avoiding those states in the first place. This triggers a cycle where the agent revisits the catastrophe, relearns the danger, and then forgets it again once it becomes safe. We term this cycle the Sisyphean Curse.

The Intrinsic Fear (IF) Framework

To combat this curse, we propose Intrinsic Fear (IF), an algorithm that augments a standard DQN with a persistent, supervised "fear model" ($F$). This model acts as a guardian, ensuring that the knowledge of danger remains a constant part of the agent's objective, regardless of its current policy or the contents of its recent experience replay.

1. The Fear Model and Persistent Memory

The fear model is a binary classifier trained to predict the probability that a given state will lead to a catastrophe within a fixed number of steps, known as the fear radius ($k_r$).

Unlike the DQN's primary replay buffer, the IF system utilizes two specialized, permanent buffers:

  • Danger Buffer: Stores states that lead directly to a catastrophe.
  • Safe Buffer: Stores states that are known to be far from any catastrophe.

Because these buffers do not use a sliding window, the fear model never "forgets" what danger looks like, providing a stable foundation for safety even when the agent is performing optimally.

2. Reward Shaping via Fear

The output of the fear model is used to penalize the Q-learning objective. We modify the standard Temporal Difference (TD) target by subtracting a "fear factor" ($\lambda$) multiplied by the fear model’s prediction:

$$y_t^{IF} = r_t + \gamma \max_{a'} Q(s_{t+1}, a') - \lambda F(s_{t+1})$$

This reward shaping provides a dense and consistent signal. By recognizing the "danger zone" (states within $k_r$ steps of a crash), the agent receives a penalty early enough to course-correct, rather than only receiving a sparse negative reward at the moment of failure. To maintain stability during the early stages of training, the fear factor $\lambda$ is gradually phased in.

Theoretical Foundations

A critical aspect of IF is its theoretical robustness. The paper provides proofs demonstrating that:

  • Return Bounds: There is a quantifiable bound on the reduction in average return when learning on the perturbed IF objective compared to the original objective.
  • Error Robustness: The resulting policy remains nearly optimal even if the fear model is imperfect, provided that the classification errors are bounded. This ensures that small inaccuracies in the fear model do not lead to catastrophic policy failures.

Experimental Evaluation

We evaluated the IF-DQN across several environments designed to test both safety and learning efficiency.

Toy Environments: Adventure Seeker and Cart-Pole

In Adventure Seeker—a 1D environment where agents gain rewards for climbing a hill but "die" if they overstep—standard DQNs repeatedly failed due to the Sisyphean Curse. Similarly, in Cart-Pole, standard agents periodically allowed the pole to fall after long periods of stability. In both cases, IF-DQNs learned to avoid catastrophes indefinitely after a short initial training period.

Atari Benchmarks

The IF framework was also applied to three games in the Arcade Learning Environment:

  • Freeway: This game provided the most dramatic results. Standard DQNs often fail to receive any reward because they never learn to cross the road. The IF-DQN used the "fear" of oncoming cars as a dense reward-shaping signal, which significantly accelerated the learning process and led to rapid improvement.
  • Asteroids and Seaquest: In these games, IF agents achieved higher total rewards and, in the case of Asteroids, significantly fewer catastrophes (lives lost) compared to standard DQNs.

Discussion and Limitations

While Intrinsic Fear provides a robust solution to catastrophic forgetting, it relies on a clear definition of "catastrophe" (e.g., a "life lost" signal). In many real-world applications, identifying these states without prior knowledge or an oracle remains a challenge.

Furthermore, the performance of the agent is sensitive to the fear radius ($k_r$) and fear factor ($\lambda$). If $k_r$ is too small, the agent may not have sufficient time to react to danger; if it is too large, the agent may become overly cautious or "paralyzed," avoiding potentially high-reward areas of the state space.

Conclusion

Intrinsic Fear addresses a fundamental flaw in how DRL agents handle safety-critical information. By treating safety as a supervised classification problem with persistent memory, IF-DQNs overcome the Sisyphean Curse of learning and forgetting. This approach not only improves safety but also enhances learning speed through effective reward shaping, marking a significant step toward more reliable and efficient autonomous agents.

Summary updated

Abstract

Many practical environments contain catastrophic states that an optimal agent would visit infrequently or never. Even on toy problems, Deep Reinforcement Learning (DRL) agents tend to periodically revisit these states upon forgetting their existence under a new policy. We introduce intrinsic fear (IF), a learned reward shaping that guards DRL agents against periodic catastrophes. IF agents possess a fear model trained to predict the probability of imminent catastrophe. This score is then used to penalize the Q-learning objective. Our theoretical analysis bounds the reduction in average return due to learning on the perturbed objective. We also prove robustness to classification errors. As a bonus, IF models tend to learn faster, owing to reward shaping. Experiments demonstrate that intrinsic-fear DQNs solve otherwise pathological environments and improve on several Atari games.
BibTeX
@misc{lipton2016combating,
  author = {Zachary C. Lipton and Jianfeng Gao and Lihong Li and Jianshu Chen and Li Deng},
  title = {{Combating Reinforcement Learning's Sisyphean Curse with Intrinsic Fear}},
  year = {2016},
  eprint = {1611.01211},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1611.01211},
}
Figure