Exploration by Learning Diverse Skills through Successor State Measures

Paul-Antoine Le Tolguenec, Yann Besse, Florent Teichteil-Königsbuch, Dennis G. Wilson, Emmanuel Rachelson

TL;DR

The LEADS algorithm (Learning Diverse Skills through Successor States) moves beyond standard mutual information maximization to discover truly exploratory behaviors in reinforcement learning. By leveraging Successor State Measures to estimate state occupancy, LEADS learns a set of diverse skills that exhaustively and uniformly cover complex environments without requiring any external rewards.

What Problem Does This Paper Address?

In unsupervised reinforcement learning, the goal is to have an agent discover a "repertoire" of useful skills without being told what to do. Traditionally, researchers have used Mutual Information (MI) maximization to achieve this. The idea is simple: make states and skills highly predictable from one another so that each skill "specializes" in a different part of the environment.

However, the authors point out a critical flaw: MI is an ambiguous measure for exploration. As shown in the grid maze example below, two different sets of skills can have the same MI score, but one might cover the entire maze while the other remains huddled in the center.

Figure 1: State distributions of two sets Z1 (left) and Z2 (right) of four skills each on a grid maze. Each skill’s visited states are represented by a different symbol and distributed uniformly.

The challenge is to move from merely "separating" skills to "spreading" them out to maximize state coverage.

Methods

The authors introduce LEADS, which formalizes skill discovery using Successor State Measures (SSM). An SSM is essentially a density estimator that tells you how likely you are to visit certain states given a specific policy.

The core of LEADS is a new objective function, $G(\theta)$, which replaces standard MI. It incorporates a specialized uncertainty measure, $u_t(s, z)$, designed with three properties:

  1. Exploration: Prioritize states that haven't been visited much by any previous skills.
  2. Repulsion: Push the current skills away from each other so they don't overlap.
  3. Targeting: Encourage skills to reach for new "target" states far from the starting point.

To implement this, the authors use C-Learning, a classification-based approach to estimate the SSM. The algorithm iterates through three phases: sampling trajectories, updating the SSM density estimator, and then optimizing the policy to maximize the new exploration-focused objective.

Datasets

The authors evaluated LEADS against seven state-of-the-art baselines (including DIAYN, LSD, and NGU) across two main categories:

  • Maze Navigation: Three custom 2D mazes (Easy, U-Maze, and Hard) designed to test an agent's ability to navigate through bottlenecks.
  • Robotic Control (MuJoCo): High-dimensional tasks including Fetch-Reach (7-DoF arm), Fetch-Slide (moving a puck), Hand (24-DoF robotic hand), and Finger (rotating an object).

Key Insights and Ideas

  • MI is a Proxy, Not the Goal: The paper argues that maximizing MI is often just a heuristic for diversity. By explicitly targeting state occupancy via SSMs, LEADS provides a more direct path to thorough exploration.
  • The Power of Repulsion: A key "aha moment" is the use of repulsion between skills. By treating skill discovery as a "probability mass transport" problem, LEADS ensures that as one skill discovers a new area, it effectively "pushes" other skills to look elsewhere.
  • Successor States as Density Estimators: Using SSMs allows the agent to understand its own "footprint" in the environment more accurately than simple count-based methods, especially in continuous state spaces.

Key Results

LEADS demonstrated superior state coverage across nearly all benchmarks:

  • Hard Maze: LEADS was the only algorithm capable of reaching all sections of the Hard maze, navigating through difficult bottlenecks that stopped other agents.
  • Fetch-Slide: In this challenging robotic task, LEADS exceeded the coverage of all other methods by over 30%.
  • Visual Diversity: As seen in the skill visualizations, LEADS produces clearly distinct, colorful "fans" of behavior that reach the edges of the state space, whereas baselines like DIAYN often collapse or cover less ground.
Figure 2: Skill visualisation for each algorithm. Per algorithm, the tasks are the mazes Easy (top left), U (top right), Hard (bottom left), and the control task Fetch-Reach (bottom right).

Limitations and Future Work

The primary limitation of LEADS is its dependence on the quality of the SSM estimator. The authors noted failure cases in environments like HalfCheetah, where the underlying C-Learning estimator struggled to generalize in high-dimensional spaces.

Future work will likely focus on integrating more robust SSM estimators (like those based on contrastive learning) to make LEADS more stable across a wider variety of complex MuJoCo environments. The authors also suggest that advances in reliable SSM estimation will directly and immediately improve the performance of the LEADS framework.

Summary updated

Abstract

The ability to perform different skills can encourage agents to explore. In this work, we aim to construct a set of diverse skills which uniformly cover the state space. We propose a formalization of this search for diverse skills, building on a previous definition based on the mutual information between states and skills. We consider the distribution of states reached by a policy conditioned on each skill and leverage the successor state measure to maximize the difference between these skill distributions. We call this approach LEADS: Learning Diverse Skills through Successor States. We demonstrate our approach on a set of maze navigation and robotic control tasks which show that our method is capable of constructing a diverse set of skills which exhaustively cover the state space without relying on reward or exploration bonuses. Our findings demonstrate that this new formalization promotes more robust and efficient exploration by combining mutual information maximization and exploration bonuses.
BibTeX
@inproceedings{tolguenec2024exploration,
  author = {Paul-Antoine Le Tolguenec and Yann Besse and Florent Teichteil-Königsbuch and Dennis G. Wilson and Emmanuel Rachelson},
  title = {{Exploration by Learning Diverse Skills through Successor State Measures}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=oyiBLfNJvY},
}
Figure