Latent Skill Discovery through Mutual Information Maximization

This direction is merged into another direction: mutual information maximization (mutual information-based skill discovery). Its last version stays readable and citable here.

Version 1 of 115 papers21 conceptsfrozen 2026-10-04

Standard reinforcement learning (RL) is often a race to the bottom—or rather, a race to a single global optimum. When we train an agent to solve a maze, we typically reward it for reaching the goal as fast as possible. The result is a "collapsed" policy: the agent learns one specific path and ignores all others.

However, in complex real-world scenarios, a single optimal path is rarely enough. We might want an agent that can solve a task with different "styles" (e.g., walking vs. running), navigate around unexpected obstacles, or extract useful sub-skills from a messy dataset of human demonstrations. To achieve this, a growing body of research focuses on Mutual Information (MI) maximization to force agents to discover a diverse library of distinct, interpretable behaviors.

The Problem of Policy Collapse

In a standard RL setup, an agent's policy $\pi(a|s)$ maps a state to an action. To introduce variety, researchers often use a latent-conditioned policy $\pi(a|s, z)$, where $z$ is a "style" or "skill" code.

The danger is that the agent might simply learn to ignore $z$. If one specific strategy is slightly more efficient than others, the agent will converge to that strategy regardless of the input code. This is known as representation collapse. To the agent, $z$ becomes "noise" that it filters out to maximize the external reward.

Mutual Information as a Diversity Engine

To prevent this collapse, researchers like Aadil Hayat, Utsav Singh, and Vinay P. Namboodiri in InfoRL propose a simple but powerful intuition: If I tell you to perform "Skill A," an observer should be able to look at your actions and guess that you were performing "Skill A."

Mathematically, this is framed as maximizing the Mutual Information $I(z; \tau)$ between the latent code $z$ and the resulting trajectory $\tau$. If the MI is high, the latent code $z$ is highly predictive of the behavior.

This is typically implemented using a "posterior" or "discriminator" network. While the agent tries to solve the task, a second network tries to reconstruct $z$ from the agent's state-action pairs. The agent receives an intrinsic "posterior reward" based on how well the discriminator can identify its current "style." This forces the agent to make its behaviors as distinct and identifiable as possible.

Disentangling Skills from Messy Data

This mechanism is particularly effective when the "correct" way to solve a task isn't clear. In DiveOff, Takayuki Osa and Tatsuya Harada apply this to offline RL. When learning from a static dataset that contains multiple ways to solve a task, MI maximization ensures the policy remains sensitive to the latent inputs, allowing it to recover the full spectrum of solutions present in the data rather than just the most frequent one.

Similarly, in the context of imitation learning, Tianxiang Zhao and colleagues developed SDIL. By maximizing MI between latent skills and state-action transitions, they can "disentangle" high-quality action primitives even from suboptimal or noisy demonstrations. The MI objective acts as a filter, forcing the model to categorize behaviors into non-overlapping, reusable skills.

Generalization and Safety

Beyond just looking "cool" or being interpretable, diversity is a functional requirement for robustness. Allen Z. Ren and Jaime Fernández Fisac demonstrate in their Sim-to-Lab-to-Real framework that an agent with a diverse set of strategies is more resilient. If a robot encounters a novel environmental layout in the real world that blocks its "primary" path, a policy distribution trained via MI maximization is more likely to contain an alternative "style" that can still succeed.

In multi-agent settings, Shenao Zhang and Lei Han use MI in MRA to separate "game-common" skills from "game-specific" tactics. This allows agents to adapt to new environments with different numbers of players by drawing from a diverse library of learned meta-representations.

Measuring Empowerment

Finally, this line of research connects to the concept of Empowerment. In Latent-Predictive Empowerment, Alessandro Allievi and George Konidaris show that maximizing MI is equivalent to maximizing an agent's potential influence over its future state. By combining MI with a penalty for "outcome redundancy"—ensuring that different action sequences don't just end up in the same place—agents can learn to master their environment entirely through intrinsic motivation, without needing a predefined reward signal or a simulator.

Summary of the Intuition

The core idea across these papers is that information is a proxy for utility. By treating the agent's behavior as a communication channel for a latent "intent" ($z$), we can use the tools of information theory to ensure that:

  1. Diversity: Different intents lead to visibly different behaviors.
  2. Interpretability: We can "read" the agent's intent by observing its actions.
  3. Robustness: The agent possesses a library of strategies rather than a single point of failure.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_latent_skill_discovery_through_mutual_in_f068dc07,
  title = {Latent Skill Discovery through Mutual Information Maximization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-skill-discovery-through-mutual-information-maximization/rdr_f068dc073185478798e8c4f641361f16},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_latent_skill_discovery_through_mutual_in_f068dc07_v1,
  title = {Latent Skill Discovery through Mutual Information Maximization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/latent-skill-discovery-through-mutual-information-maximization/rdr_f068dc073185478798e8c4f641361f16/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}