Exploration by Learning Diverse Skills through Successor State Measures

Paul-Antoine Le Tolguenec, Yann Besse, Florent Teichteil-Königsbuch, Dennis George Wilson, Emmanuel Rachelson

TL;DR

This paper introduces LEADS, a novel algorithm for reinforcement learning that learns a diverse set of skills to efficiently explore an environment without relying on external rewards. By leveraging successor state measures and a new uncertainty-based objective, LEADS effectively balances skill distinctiveness with comprehensive state coverage, outperforming existing methods on various benchmarks.

What Problem Does This Paper Address?

In reinforcement learning (RL), agents often need to explore their environment to discover optimal policies. A key challenge is how to encourage effective exploration, especially in complex, high-dimensional environments where rewards might be sparse or non-existent. While learning diverse skills has been shown to promote exploration, existing methods often struggle to balance skill distinctiveness with broad state coverage. Specifically, approaches based on maximizing mutual information between states and skills can be ambiguous, failing to differentiate between skills that cover a small, distinct set of states versus those that cover a larger, more diverse portion of the environment. This paper addresses this gap by proposing a new objective that explicitly promotes both skill diversity and comprehensive state coverage, leading to more robust and efficient exploration.

Methods

The core of LEADS lies in its novel objective function, which is derived from the concept of mutual information but modified to encourage exploration. Instead of directly maximizing mutual information, LEADS leverages Successor State Measures (SSMs) to estimate state distributions for each skill. SSMs provide a way to estimate the expected future state occupancy given any starting state and a specific skill.

The proposed objective function, denoted as $G(\theta)$, combines two key components:

  1. Exploration Term: This term uses an uncertainty measure ($u_t(s, z)$) to identify states that are both likely to be visited by a skill and have been infrequently visited by other skills or previous policies. This encourages the agent to explore under-visited areas.
  2. Repulsion Term: This term aims to ensure that the state distributions induced by different skills are distinct, preventing skills from collapsing into similar behaviors. It leverages Kullback-Leibler (KL) divergences between skill-specific state distributions.

The uncertainty measure $u_t(s, z)$ is designed to capture both the novelty of a state for a specific skill (exploration) and its distinctiveness from other skills (diversity). LEADS then uses this uncertainty measure to define a sampling distribution $\delta(s|z)$ that biases exploration towards promising states. The algorithm iteratively learns the SSMs for each skill and updates the policy parameters to maximize $G(\theta)$, effectively learning a set of diverse skills that cover the state space.

Datasets

The paper evaluates LEADS on a variety of standard RL benchmarks, including:

  • Maze Navigation Tasks: Three mazes of increasing difficulty (Easy, U, Hard) with a 2D state space.
  • Robotic Control Tasks: Several MuJoCo environments, including Fetch-Reach, Fetch-Slide, and Finger, which are high-dimensional and complex.

No new datasets were introduced; the focus was on evaluating the algorithm's performance on existing, challenging environments.

Key Insights and Ideas

  • Mutual Information Ambiguity: The paper highlights that simply maximizing mutual information between states and skills isn't sufficient for promoting effective exploration, as it can favor skills that cover small, distinct regions over those that cover larger, more diverse areas.
  • SSMs as a Foundation: Successor State Measures are crucial for estimating skill-specific state distributions, providing a principled way to quantify skill behavior.
  • Uncertainty for Exploration and Diversity: The novel uncertainty measure $u_t(s, z)$ effectively combines the goals of exploring under-visited states and ensuring skills remain distinct, a key innovation.
  • Reward-Free Exploration: LEADS demonstrates that robust exploration can be achieved without relying on extrinsic rewards or exploration bonuses, making it applicable to a wider range of RL problems.
  • Skill Specialization: The algorithm learns to specialize each skill towards specific states that maximize the uncertainty measure, leading to a more structured and efficient exploration process.

Key Results

LEADS consistently outperforms several state-of-the-art exploration and skill discovery algorithms across various tasks.

  • Maze Navigation: LEADS achieves superior state coverage, particularly in the challenging "Hard" maze, where it is the only algorithm to reach all sections.
  • Robotic Control: In high-dimensional environments like Fetch-Slide, LEADS significantly exceeds the coverage of other methods by over 30%. It also demonstrates strong performance in Fetch-Reach, achieving 90% coverage compared to other methods' around 80%.
  • Generalization: While LEADS shows strong performance on many tasks, it is noted that its performance is intrinsically tied to the quality of the SSM estimator. In some complex MuJoCo environments, the chosen SSM estimator (C-Learning) struggled, leading to suboptimal results.

Limitations and Future Work

  • SSM Estimator Dependency: The paper acknowledges that LEADS's performance is heavily reliant on the quality of the Successor State Measure (SSM) estimator. Poor SSM estimation, particularly in high-dimensional MuJoCo environments, can limit LEADS's effectiveness.
  • C-Learning Limitations: The authors note that C-Learning, the SSM estimator used in their experiments, faced challenges in certain MuJoCo environments. They suggest that advances in reliable SSM estimation, especially those demonstrating improved generalization capabilities, could further benefit LEADS.
  • Future Work: The authors reserve the extension of LEADS to utilize more advanced SSM estimators for future work, aiming to improve performance in challenging, high-dimensional domains. They also suggest that further research into robust SSM estimation will directly benefit and open new avenues for LEADS.

Summary updated

Abstract

The ability to perform different skills can encourage agents to explore. In this work, we aim to construct a set of diverse skills that uniformly cover the state space. We propose a formalization of this search for diverse skills, building on a previous definition based on the mutual information between states and skills. We consider the distribution of states reached by a policy conditioned on each skill and leverage the successor state measure to maximize the difference between these skill distributions. We call this approach LEADS: Learning Diverse Skills through Successor State Measures. We demonstrate our approach on a set of maze navigation and robotic control tasks which show that our method is capable of constructing a diverse set of skills which exhaustively cover the state space without relying on reward or exploration bonuses. Our findings demonstrate that this new formalization promotes more robust and efficient exploration by combining mutual information maximization and exploration bonuses.
BibTeX
@inproceedings{tolguenec2024exploration,
  author = {Paul-Antoine LE TOLGUENEC and Yann Besse and Florent Teichteil-Königsbuch and Dennis George Wilson and Emmanuel Rachelson},
  title = {{Exploration by Learning Diverse Skills through Successor State Measures}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=oyiBLfNJvY},
}
Figure