Introduction
Reinforcement learning (RL) agents often struggle in environments with "sparse rewards," where meaningful feedback is rare. Standard exploration techniques, such as $\epsilon$-greedy noise, are frequently insufficient for complex tasks like robotic locomotion or strategic games. This paper proposes a scalable framework for intrinsic motivation based on "surprise"—rewarding agents for encountering transitions that their internal dynamics models fail to predict accurately.
Methodology
The core of the approach is the concurrent training of a dynamics model $P_\phi(s'|s, a)$ alongside the agent's policy. This model, implemented as a neural network, predicts the next state $s'$ given the current state $s$ and action $a$. The authors introduce two primary intrinsic reward signals:
- Surprisal (NLL): The negative log-likelihood of the observed transition under the current model. This directly rewards the agent for encountering unexpected outcomes.
- $k$-step Learning Progress (AKL): The difference in prediction quality between the current model and a version from $k$ updates ago, rewarding the agent for exploring areas where its predictive ability is improving most rapidly.
By using Trust Region Policy Optimization (TRPO) as the base algorithm and a fully-factored Gaussian dynamics model, the system remains computationally efficient and applicable to both continuous and discrete state spaces.
Experimental Evaluation
The proposed methods were tested on two challenging suites:
- MuJoCo Continuous Control: Sparse-reward versions of classic tasks (MountainCar, HalfCheetah) and a new complex task, SwimmerGather, requiring both locomotion and navigation.
- Atari RAM: Difficult exploration games like Venture and Freeway, using raw RAM states.
Key Results
The surprise-based incentives significantly outperformed naive TRPO, which often failed to achieve any reward in sparse environments. In MuJoCo tasks, both NLL and AKL variants matched or exceeded the performance of more complex Bayesian methods like VIME.
A critical advantage of the surprisal-based approach is its computational efficiency. Unlike previous state-of-the-art methods like Variational Information Maximizing Exploration (VIME), which require expensive Bayesian updates, the surprisal bonus relies on a simple forward pass, resulting in a significant speedup in both initialization and iteration time.
In Atari environments, the method demonstrated robust exploration. For instance, in Venture-RAM, the surprisal incentive led to scores exceeding 500, whereas standard RL agents typically fail to find any reward.
Discussion and Conclusion
The results indicate that neural network generalization serves as a powerful exploration tool: as the model learns to predict familiar transitions, the resulting "breadcrumb trail" of high prediction error naturally guides the agent toward novel regions. While challenges remain—such as sensitivity to the "noisy TV" problem and hyperparameter tuning—this work provides a practical, high-performance path toward solving sparse-reward tasks in high-dimensional spaces.
Summary updated