Unsupervised Skill Discovery through Mutual Information

This direction is merged into another direction: mutual information maximization (mutual information-based skill discovery). Its last version stays readable and citable here.

Version 1 of 190 papers106 conceptsfrozen 2026-10-04

Reinforcement learning (RL) typically relies on a reward function to tell an agent what to do. However, in the real world, rewards are often sparse or non-existent. A robot might spend hours wandering a room without ever receiving a "point." To solve this, researchers have turned to unsupervised skill discovery, where agents learn a library of useful behaviors—like walking, jumping, or pushing objects—without any external supervision.

The central mechanism driving this discovery is the maximization of Mutual Information (MI) between a latent "skill" variable and the agent's behavior.

The Intuition of Distinguishability

Imagine you give a robot a random number (a latent code $z$) and tell it to "do something" associated with that number. If the robot performs the exact same motion regardless of whether you gave it $z=1$ or $z=2$, those numbers carry no information. However, if the robot walks forward for $z=1$ and jumps for $z=2$, the number and the behavior are highly correlated.

This is the core idea behind Diversity is All You Need (DIAYN) by Benjamin Eysenbach, Sergey Levine, and colleagues. They framed skill discovery as a game between two components:

  1. The Agent, which tries to act in a way that makes its current skill $z$ easy to identify.
  2. The Discriminator, a separate model that looks at the agent’s state and tries to guess which $z$ the agent was following.

By maximizing the mutual information $I(S; Z)$, the agent is incentivized to visit states that are unique to a specific skill. If the discriminator can easily guess the skill from the state, the agent has successfully learned a distinct, reproducible behavior.

Beyond Diversity to Predictability

While DIAYN creates diverse behaviors, it doesn't guarantee they are useful or controllable. An agent might discover a "skill" that involves twitching randomly in a corner—diverse, but not helpful for downstream tasks.

To address this, Archit Sharma and Karol Hausman introduced Dynamics-Aware Unsupervised Discovery of Skills (DADS). Instead of just making skills distinguishable, DADS maximizes the mutual information between the skill and the next state, conditioned on the current state. This forces the agent to learn skills that are not only diverse but also predictable. If the agent knows exactly how a skill will change the environment, it can sequence those skills together to solve new tasks "zero-shot" through planning.

Balancing Exploration and Skill Acquisition

A common failure mode in MI-based discovery is that an agent might learn many distinct skills that all stay within a tiny, safe corner of the environment. To truly master a space, the agent needs to explore.

Hao Liu and Pieter Abbeel showed in APS: Active Pretraining with Successor Features that the MI objective can be decomposed into two parts:

  • State Entropy: Encouraging the agent to visit as many different states as possible (exploration).
  • Conditional Entropy: Ensuring that, given a specific skill, the resulting states are predictable (distinguishability).

By optimizing both, agents like those in APS and LEADS achieve broad environmental coverage while still building a structured library of behaviors.

Modern Refinements: Contrastive Learning and Disentanglement

Recent work has moved toward making these objectives more stable and scalable. Chongyi Zheng and Jens Tuyls argued in Can a MISL Fly? that many high-performing algorithms are actually performing contrastive MI maximization. By pushing the representations of different skills apart in a latent space, algorithms like Contrastive Successor Features (CSF) achieve state-of-the-art results without the complex dual optimization required by earlier methods.

Furthermore, researchers are looking at how to make these skills more modular. Jiaheng Hu and Peter Stone developed DUSDi, which uses MI to learn disentangled skills. Instead of one skill that moves a robot forward and turns its head simultaneously, DUSDi encourages skills that control independent factors of the environment—allowing a high-level controller to "mix and match" behaviors more effectively.

Why It Works

Unsupervised skill discovery works because it treats "learning" as a process of organizing the state space. By maximizing mutual information, the agent is essentially building a map where every point (latent code) corresponds to a reliable, distinct physical maneuver. When a human finally provides a task, the agent doesn't have to learn how to move from scratch; it simply learns which of its pre-existing skills to call upon.

Papers in this version

Showing 60 of 90 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unsupervised_skill_discovery_through_mut_118929c9,
  title = {Unsupervised Skill Discovery through Mutual Information},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unsupervised-skill-discovery-through-mutual-information/rdr_118929c91fa543e5852a780627e6011b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_unsupervised_skill_discovery_through_mut_118929c9_v1,
  title = {Unsupervised Skill Discovery through Mutual Information},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unsupervised-skill-discovery-through-mutual-information/rdr_118929c91fa543e5852a780627e6011b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}