Planning to Explore via Self-Supervised World Models

Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, Deepak Pathak

In the quest for artificial general intelligence, one of the biggest hurdles is the "one-task-at-a-time" nature of traditional reinforcement learning (RL). Usually, we define a specific reward (like "don't fall over"), and the agent spends millions of steps learning that single skill. If you want it to jump instead of walk, you often have to start from scratch.

Plan2Explore, a paper from ICML 2020 by Sekar et al., proposes a more human-like alternative: let the agent play and explore the world first without any specific goal, building a "world model" of how things work. Once it understands the physics of its environment, it can learn new tasks almost instantly.

TL;DR

Plan2Explore is a self-supervised RL agent that learns a robust world model by proactively planning to visit states where its internal predictions are most uncertain (a concept called "latent disagreement"). Once trained, this single model can solve multiple downstream tasks—like walking, running, or flipping—in a zero-shot or few-shot manner, often matching the performance of agents trained with task-specific rewards from the start.

Overview of the Plan2Explore framework, showing the transition from task-agnostic exploration to zero-shot adaptation.

What Problem Does This Paper Address?

Traditional RL is sample inefficient and task-specific. If you change the goal, you usually need massive amounts of new data.

While "intrinsic motivation" (curiosity) has been used to help agents explore, most prior methods are retrospective. They calculate how "surprising" a state was after the agent already arrived there. This is like wandering aimlessly until you stumble upon something interesting.

The authors argue that an intelligent agent should be proactive: it should use its internal model of the world to plan a path toward future situations it expects will be novel or informative.

Methods

The core of Plan2Explore is a two-stage process:

  1. Self-Supervised Exploration: The agent interacts with an environment without any rewards. It uses a Recurrent State Space Model (RSSM) to learn latent representations of the world directly from pixels.
  2. Latent Disagreement: To decide where to go, the agent maintains an ensemble of small neural networks that try to predict the next latent state. If the networks in the ensemble disagree with each other, it means the agent is uncertain about that area.
  3. Planning to Explore: Unlike "reactive" curiosity, Plan2Explore uses its world model to imagine future trajectories and picks the actions that lead to the highest expected disagreement (uncertainty) among the ensemble.
  4. Task Adaptation: Once the exploration phase is over, the agent is given a reward function for a specific task (e.g., "move forward as fast as possible").
  5. Zero-Shot: The agent trains a policy entirely in its "imagination" using the world model, without any further environment interaction.
  6. Few-Shot: If the zero-shot policy isn't perfect, the agent performs a small amount of task-specific fine-tuning.

Datasets

The researchers evaluated Plan2Explore on the DeepMind Control Suite, a standard benchmark for continuous control. They tested across 20 challenging tasks involving different simulated robots (Acrobot, Hopper, Cheetah, Walker, etc.). Crucially, the agent only "sees" raw 64x64 pixels—it has no direct access to the robot's joint positions or velocities.

Key Insights and Ideas

  • Proactive vs. Reactive: The biggest "aha" moment is that planning for future novelty is significantly more efficient than reacting to past novelty. By "dreaming" about where it might be confused, the agent targets its exploration much more effectively.
  • The Power of Disagreement: The paper provides a theoretical link showing that maximizing the variance (disagreement) of an ensemble approximates maximizing Expected Information Gain. This grounds a heuristic "curiosity" measure in formal information theory.
  • Global vs. Local Models: A supervised agent trained only to "run forward" learns a "local" model—it only understands the physics relevant to running forward. Plan2Explore learns a "global" model. Because it explored everywhere, it understands the physics of the whole environment, making it much more versatile.

Key Results

  • Zero-Shot Success: In many tasks, Plan2Explore achieved state-of-the-art zero-shot performance. For example, after exploring the "Cheetah" environment without rewards, it could learn to run forward, run backward, or flip, purely in imagination.
  • Matching the Oracle: Plan2Explore often matched the performance of a "Supervised Oracle" (the Dreamer algorithm)—an agent that had access to rewards during the entire training process.
  • Superior Exploration: It consistently outperformed other popular exploration methods like Curiosity (Pathak et al.) and MAX (Shyam et al.), particularly in high-dimensional visual environments.
Zero-shot performance of Plan2Explore compared to baselines across various continuous control tasks.

Limitations and Future Work

  • Model Bottleneck: The agent's performance is strictly capped by the quality of its world model. If the environment is too complex for the RSSM to capture (e.g., highly stochastic or visually overwhelming), exploration will suffer.
  • Computational Overhead: Maintaining an ensemble of models adds computational cost, though the authors note these models are lightweight and can be trained in parallel.
  • Real-World Transfer: While successful in simulation, the "real world" has infinite complexity. Future work involves testing how these self-supervised models handle the messy, non-deterministic physics of actual robotic hardware.

By shifting the focus from "learning a task" to "learning the world," Plan2Explore moves us closer to agents that can walk into a new room, look around, and immediately know how to be useful.

Summary updated

Abstract

Reinforcement learning allows solving complex tasks, however, the learning tends to be task-specific and the sample efficiency remains a challenge. We present Plan2Explore, a self-supervised reinfo...
BibTeX
@inproceedings{sekar2020planning,
  author = {Ramanan Sekar and Oleh Rybkin and Kostas Daniilidis and Pieter Abbeel and Danijar Hafner and Deepak Pathak},
  title = {{Planning to Explore via Self-Supervised World Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=9g-UWHVorL},
}
Figure