Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning

Joshua Achiam, Shankar Sastry

Introduction

Reinforcement learning (RL) agents often struggle in environments with "sparse rewards," where meaningful feedback is rare. Standard exploration techniques, such as $\epsilon$-greedy noise, are frequently insufficient for complex tasks like robotic locomotion or strategic games. This paper proposes a scalable framework for intrinsic motivation based on "surprise"—rewarding agents for encountering transitions that their internal dynamics models fail to predict accurately.

Methodology

The core of the approach is the concurrent training of a dynamics model $P_\phi(s'|s, a)$ alongside the agent's policy. This model, implemented as a neural network, predicts the next state $s'$ given the current state $s$ and action $a$. The authors introduce two primary intrinsic reward signals:

  1. Surprisal (NLL): The negative log-likelihood of the observed transition under the current model. This directly rewards the agent for encountering unexpected outcomes.
  2. $k$-step Learning Progress (AKL): The difference in prediction quality between the current model and a version from $k$ updates ago, rewarding the agent for exploring areas where its predictive ability is improving most rapidly.

By using Trust Region Policy Optimization (TRPO) as the base algorithm and a fully-factored Gaussian dynamics model, the system remains computationally efficient and applicable to both continuous and discrete state spaces.

Experimental Evaluation

The proposed methods were tested on two challenging suites:

  • MuJoCo Continuous Control: Sparse-reward versions of classic tasks (MountainCar, HalfCheetah) and a new complex task, SwimmerGather, requiring both locomotion and navigation.
  • Atari RAM: Difficult exploration games like Venture and Freeway, using raw RAM states.

Key Results

The surprise-based incentives significantly outperformed naive TRPO, which often failed to achieve any reward in sparse environments. In MuJoCo tasks, both NLL and AKL variants matched or exceeded the performance of more complex Bayesian methods like VIME.

Performance comparison on MuJoCo continuous control tasks. Surprisal (NLL) and Learning Progress (AKL) variants consistently outperform standard TRPO and match or exceed VIME.

A critical advantage of the surprisal-based approach is its computational efficiency. Unlike previous state-of-the-art methods like Variational Information Maximizing Exploration (VIME), which require expensive Bayesian updates, the surprisal bonus relies on a simple forward pass, resulting in a significant speedup in both initialization and iteration time.

Computational efficiency comparison. The surprisal-based approach (NLL) offers a significant reduction in both initialization and per-iteration time compared to VIME.

In Atari environments, the method demonstrated robust exploration. For instance, in Venture-RAM, the surprisal incentive led to scores exceeding 500, whereas standard RL agents typically fail to find any reward.

Discussion and Conclusion

The results indicate that neural network generalization serves as a powerful exploration tool: as the model learns to predict familiar transitions, the resulting "breadcrumb trail" of high prediction error naturally guides the agent toward novel regions. While challenges remain—such as sensitivity to the "noisy TV" problem and hyperparameter tuning—this work provides a practical, high-performance path toward solving sparse-reward tasks in high-dimensional spaces.

Summary updated

BibTeX
@inproceedings{achiam2017surprise,
  author = {Joshua Achiam and Shankar Sastry},
  title = {{Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2017},
  url = {https://openreview.net/forum?id=Bk8aOm9xl},
}
Figure