Self-Supervised Exploration via Disagreement

Deepak Pathak, Dhiraj Gandhi, Abhinav Gupta

Abstract

Efficient exploration is a long-standing problem in sensorimotor learning. Major advances have been demonstrated in noise-free, non-stochastic domains such as video games and simulation. However, most of these formulations either get stuck in environments with stochastic dynamics or are too inefficient to be scalable to real robotics setups. In this paper, we propose a formulation for exploration inspired by the work in active learning literature. Specifically, we train an ensemble of dynamics models and incentivize the agent to explore such that the disagreement of those ensembles is maximized. This allows the agent to learn skills by exploring in a self-supervised manner without any external reward. Notably, we further leverage the disagreement objective to optimize the agent's policy in a differentiable manner, without using reinforcement learning, which results in a sample-efficient exploration. We demonstrate the efficacy of this formulation across a variety of benchmark environments including stochastic-Atari, Mujoco and Unity. Finally, we implement our differentiable exploration on a real robot which learns to interact with objects completely from scratch.

Introduction

In reinforcement learning (RL), agents often struggle in environments where external rewards are sparse or non-existent. To overcome this, researchers use "intrinsic motivation" or "curiosity," rewarding agents for visiting states that are novel or hard to predict. However, standard curiosity formulations typically rely on prediction error—the difference between what the agent thought would happen and what actually occurred. This leads to the "Noisy TV" problem: an agent becomes "addicted" to random noise (like a TV showing static) because it is perpetually unpredictable, even though it contains no learnable information.

This paper introduces a novel intrinsic reward based on the disagreement (variance) of an ensemble of forward dynamics models. This approach allows agents to ignore unlearnable noise and explore more efficiently, even enabling real-world robots to learn object interaction from scratch.

Methodology: Exploration via Disagreement

The core idea is to shift from measuring how much a single model fails to predict the future to measuring how much an ensemble of models disagree with each other.

The proposed architecture uses an ensemble of dynamics models where the intrinsic reward is the variance (disagreement) of their predictions.

Ensemble of Dynamics Models

The agent maintains multiple forward models $\{f_1, f_2, ..., f_n\}$. Each model is trained to predict the next state (in a feature space) given the current state and action. The intrinsic reward is defined as the variance across the outputs of these models.

Handling Stochasticity

In a noisy environment, all models in the ensemble will eventually converge to predicting the mean of the noise. Once they converge to the same mean, their variance (disagreement) drops to zero, and the agent loses interest. In truly novel states, the models (initialized differently) will produce wildly different predictions, yielding a high reward. This distinguishes "unlearnable" randomness from "unvisited" novelty.

Differentiable Exploration

Because the reward is a function of the models' own outputs and does not depend on the ground-truth next state from the environment, the authors can backpropagate gradients directly from the disagreement objective through the forward models to the policy. This allows for "mental simulation" and significantly higher sample efficiency compared to standard RL methods like PPO.

Experimental Results

The disagreement-based approach was evaluated across diverse benchmarks, including standard and stochastic Atari games, 3D navigation in Unity, and robotic manipulation.

Comparison on Unity environments. In the presence of a "Noisy TV" (UnityTV), the disagreement-based agent (green) successfully ignores the noise and reaches the goal, while prediction-error methods (blue) are significantly slowed down.

Key Findings

  • Beating the Noisy TV: In Unity environments with random noise, the disagreement agent successfully ignored the TV to reach the goal, whereas prediction-error agents got stuck staring at the screen.
  • Robustness to Noise: On "Sticky" Atari games, the proposed method outperformed state-of-the-art curiosity methods like RND or ICM, which were hampered by environmental randomness.
  • Real-World Efficiency: The differentiable version allowed a real Sawyer robot to learn object interaction in less than 1,000 interactions. For comparison, standard RL methods often require tens of thousands of samples for similar tasks.
  • Generalization: After exploring with 20 training objects, the robot interacted effectively with 10 unseen objects, achieving a 67% interaction rate compared to 17% for a random policy.

Discussion and Conclusion

The shift from prediction error to ensemble uncertainty targets the core weakness of curiosity-based exploration. By making the objective differentiable, the agent can optimize its policy using supervised-learning-like gradients, which is significantly faster than traditional RL.

While the method currently works best for short-horizon tasks and depends on quality feature representations, it provides a robust and sample-efficient path forward for self-supervised learning in complex, stochastic environments.

Summary updated

Abstract

Efficient exploration is a long-standing problem in sensorimotor learning. Major advances have been demonstrated in noise-free, non-stochastic domains such as video games and simulation. However, most of these formulations either get stuck in environments with stochastic dynamics or are too inefficient to be scalable to real robotics setups. In this paper, we propose a formulation for exploration inspired by the work in active learning literature. Specifically, we train an ensemble of dynamics models and incentivize the agent to explore such that the disagreement of those ensembles is maximized. This allows the agent to learn skills by exploring in a self-supervised manner without any external reward. Notably, we further leverage the disagreement objective to optimize the agent's policy in a differentiable manner, without using reinforcement learning, which results in a sample-efficient exploration. We demonstrate the efficacy of this formulation across a variety of benchmark environments including stochastic-Atari, Mujoco and Unity. Finally, we implement our differentiable exploration on a real robot which learns to interact with objects completely from scratch. Project videos and code are at https://pathak22.github.io/exploration-by-disagreement/
BibTeX
@inproceedings{pathak2019self,
  author = {Deepak Pathak and Dhiraj Gandhi and Abhinav Gupta},
  title = {{Self-Supervised Exploration via Disagreement}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=H1VnZsWdWS},
}
Figure