TL;DR
The LEADS algorithm (Learning Diverse Skills through Successor States) moves beyond standard mutual information maximization to discover truly exploratory behaviors in reinforcement learning. By leveraging Successor State Measures to estimate state occupancy, LEADS learns a set of diverse skills that exhaustively and uniformly cover complex environments without requiring any external rewards.
What Problem Does This Paper Address?
In unsupervised reinforcement learning, the goal is to have an agent discover a "repertoire" of useful skills without being told what to do. Traditionally, researchers have used Mutual Information (MI) maximization to achieve this. The idea is simple: make states and skills highly predictable from one another so that each skill "specializes" in a different part of the environment.
However, the authors point out a critical flaw: MI is an ambiguous measure for exploration. As shown in the grid maze example below, two different sets of skills can have the same MI score, but one might cover the entire maze while the other remains huddled in the center.
The challenge is to move from merely "separating" skills to "spreading" them out to maximize state coverage.
Methods
The authors introduce LEADS, which formalizes skill discovery using Successor State Measures (SSM). An SSM is essentially a density estimator that tells you how likely you are to visit certain states given a specific policy.
The core of LEADS is a new objective function, $G(\theta)$, which replaces standard MI. It incorporates a specialized uncertainty measure, $u_t(s, z)$, designed with three properties:
- Exploration: Prioritize states that haven't been visited much by any previous skills.
- Repulsion: Push the current skills away from each other so they don't overlap.
- Targeting: Encourage skills to reach for new "target" states far from the starting point.
To implement this, the authors use C-Learning, a classification-based approach to estimate the SSM. The algorithm iterates through three phases: sampling trajectories, updating the SSM density estimator, and then optimizing the policy to maximize the new exploration-focused objective.
Datasets
The authors evaluated LEADS against seven state-of-the-art baselines (including DIAYN, LSD, and NGU) across two main categories:
- Maze Navigation: Three custom 2D mazes (Easy, U-Maze, and Hard) designed to test an agent's ability to navigate through bottlenecks.
- Robotic Control (MuJoCo): High-dimensional tasks including Fetch-Reach (7-DoF arm), Fetch-Slide (moving a puck), Hand (24-DoF robotic hand), and Finger (rotating an object).
Key Insights and Ideas
- MI is a Proxy, Not the Goal: The paper argues that maximizing MI is often just a heuristic for diversity. By explicitly targeting state occupancy via SSMs, LEADS provides a more direct path to thorough exploration.
- The Power of Repulsion: A key "aha moment" is the use of repulsion between skills. By treating skill discovery as a "probability mass transport" problem, LEADS ensures that as one skill discovers a new area, it effectively "pushes" other skills to look elsewhere.
- Successor States as Density Estimators: Using SSMs allows the agent to understand its own "footprint" in the environment more accurately than simple count-based methods, especially in continuous state spaces.
Key Results
LEADS demonstrated superior state coverage across nearly all benchmarks:
- Hard Maze: LEADS was the only algorithm capable of reaching all sections of the Hard maze, navigating through difficult bottlenecks that stopped other agents.
- Fetch-Slide: In this challenging robotic task, LEADS exceeded the coverage of all other methods by over 30%.
- Visual Diversity: As seen in the skill visualizations, LEADS produces clearly distinct, colorful "fans" of behavior that reach the edges of the state space, whereas baselines like DIAYN often collapse or cover less ground.
Limitations and Future Work
The primary limitation of LEADS is its dependence on the quality of the SSM estimator. The authors noted failure cases in environments like HalfCheetah, where the underlying C-Learning estimator struggled to generalize in high-dimensional spaces.
Future work will likely focus on integrating more robust SSM estimators (like those based on contrastive learning) to make LEADS more stable across a wider variety of complex MuJoCo environments. The authors also suggest that advances in reliable SSM estimation will directly and immediately improve the performance of the LEADS framework.
Summary updated