Self-Supervised Exploration via Disagreement

Deepak Pathak, Dhiraj Gandhi, Abhinav Gupta

Introduction

Efficient exploration remains a long-standing challenge in reinforcement learning (RL). While major advances have been demonstrated in noise-free, non-stochastic domains, real-world environments are inherently unpredictable. Most current exploration methods rely on "intrinsic rewards" (curiosity), typically defined as the error in an agent's prediction of the next state.

However, prediction-error methods face two major hurdles:

  1. The "Noisy TV" Problem: In environments with random, unpredictable elements (like a TV flickering random noise), a curiosity-driven agent gets "stuck." Because it can never perfectly predict the random noise, it receives a constant high reward for its failure, halting meaningful exploration.
  2. Sample Inefficiency: Standard exploration methods treat the reward as a black box, relying on standard RL algorithms (like PPO or REINFORCE) to optimize the policy. These algorithms are notoriously slow, often requiring millions of interactions to learn simple behaviors.

This paper introduces a novel method for self-supervised exploration that overcomes these issues by training an ensemble of dynamics models and rewarding the agent for actions where these models disagree.

Methods: Exploration via Disagreement

Instead of relying on a single model's prediction error, the authors propose training an ensemble of several forward dynamics models.

Ensemble of dynamics models calculating intrinsic reward based on prediction variance.
  • The Variance Reward: The intrinsic reward is defined as the variance (disagreement) among the ensemble's predictions. If all models agree on the next state, the area is considered "known." If they provide wildly different predictions, the area is "novel," incentivizing the agent to explore it.
  • Handling Stochasticity: In a random environment (the Noisy TV scenario), all models in the ensemble will eventually converge to predicting the mean of the stochastic noise. Once they reach this consensus, their disagreement (variance) drops to zero. The agent subsequently loses interest in the noise, effectively solving the Noisy TV problem.
  • Differentiable Optimization: Because the reward is calculated based on the internal state of the models—rather than the actual next state of the environment—the authors can backpropagate gradients directly through the models into the agent's policy. This transforms exploration into a supervised-like optimization problem, drastically improving sample efficiency.

Key Insights

  • Variance vs. Error: Shifting from "prediction error" to "prediction variance" is crucial for ignoring uncontrollable noise. While prediction error stays high in random environments, variance disappears as models reach a consensus on the average outcome.
  • Exploration as "Mental Simulation": By calculating the disagreement reward using the models' internal predictions, the agent can "imagine" the reward for an action before taking it, enabling the use of analytic gradients.
  • Latent Feature Space: To avoid getting bogged down in pixel-level details, the models operate in a latent feature space (using random features or ImageNet-pretrained ResNets). This helps the agent focus on semantically meaningful changes rather than trivial visual noise.

Key Results

The researchers evaluated their approach across diverse benchmarks, including Atari games, Noisy MNIST, Unity 3D Navigation, and real-world robotics.

Performance comparison in Unity environments with and without a stochastic 'Noisy TV'.
  • Solving the Noisy TV: In the Unity 3D maze environment, the disagreement-based agent successfully ignored a flickering TV and explored the maze. In contrast, previous state-of-the-art curiosity methods became indefinitely stuck observing the noise.
  • Superior Efficiency: In the Atari game "Enduro," the differentiable version of the algorithm learned significantly faster than the version using standard RL (PPO).
  • Real-World Success: The system enabled a real 7-DOF Sawyer robot arm to learn how to interact with and move physical objects from scratch in fewer than 1,000 interactions—a stark contrast to the tens of thousands or millions of samples typically required by standard RL.
  • Zero-Shot Generalization: After exploring without any external rewards, the robot's learned policy interacted with 10 previously unseen objects with a 67% success rate, compared to just 17% for a random policy.

Limitations and Future Work

While highly effective, the approach has limitations. The differentiable optimization currently excels at short-term exploration; scaling this to "long-horizon" planning remains challenging due to the difficulty of training stable multi-step forward models. Additionally, the method relies heavily on ensemble diversity. If all models initialize or overfit identically, the disagreement signal may vanish prematurely. Finally, training an ensemble of $N$ models increases computational cost roughly $N$-fold, though this is often a worthwhile trade-off for the massive gains in sample efficiency.

Summary updated

Abstract

Efficient exploration is a long-standing problem in sensorimotor learning. Major advances have been demonstrated in noise-free, non-stochastic domains such as video games and simulation. However, m...
BibTeX
@inproceedings{pathak2019self,
  author = {Deepak Pathak and Dhiraj Gandhi and Abhinav Gupta},
  title = {{Self-Supervised Exploration via Disagreement}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=H1VnZsWdWS},
}
Figure