How can an artificial agent learn to navigate a maze, flip a robotic spatula, or develop stable walking gaits without ever receiving an external reward signal?
In traditional reinforcement learning, agents are entirely dependent on hand-engineered reward functions. When rewards are sparse or absent, agents wander aimlessly. Unsupervised skill discovery solves this by replacing external rewards with an information-theoretic objective. Instead of chasing a task-specific goal, the agent maximizes the mutual information between its behaviors and a latent "skill code."
By understanding the mathematical formulations of this objective, we can see exactly how abstract information theory forces an agent to develop a diverse, predictable, and highly structured library of control primitives.
The Core Intuition: Distinguishability
To discover skills without a reward function, we introduce a latent variable $Z$ (the skill code) sampled from a prior distribution $p(z)$. We then condition the agent's policy on this code, yielding $\pi(a|s, z)$.
Our goal is to ensure that choosing a different $z$ results in a fundamentally different behavior. We can formalize this by maximizing the mutual information $I(S; Z)$ between the states $S$ visited by the agent and the latent skill $Z$:
$$I(S; Z) = H(Z) - H(Z|S)$$
To build intuition, let's look at the two terms on the right-hand side:
- Maximize $H(Z)$: This is the entropy of our skill distribution. If we choose a uniform prior over a discrete set of skills, $H(Z)$ is fixed at its maximum possible value. This forces the agent to utilize all available skill codes equally rather than collapsing them into a single behavior.
- Minimize $H(Z|S)$: This is the conditional entropy of the skill given the state. It answers the question: If an observer looks at a state visited by the agent, how uncertain are they about which skill code generated that state? Minimizing this means reducing uncertainty to zero.
In other words, maximizing mutual information means making the agent's behaviors perfectly distinguishable.
Latent Skill (Z) ───> Policy π(a|s,z) ───> Trajectory (S) ───> Discriminator q(z|s) ───> Predicted Skill (Ẑ)
│
[Maximize Accuracy]
In practice, calculating the true posterior $p(z|s)$ is intractable because it depends on the environment's unknown dynamics. As shown in foundational frameworks like Variational Intrinsic Control by Karol Gregor et al. and Stochastic Neural Networks for Hierarchical Reinforcement Learning by Carlos Florensa et al., we can optimize a variational lower bound instead.
We train an auxiliary neural network—a discriminator $q_\phi(z|s)$—to guess the active skill $z$ from the visited states. The agent is then given an intrinsic reward proportional to $\log q_\phi(z|s)$. As popularized by Diversity is All You Need: Learning Skills without a Reward Function (Abhishek Gupta et al.), this simple game forces the policy to partition the environment's state space into distinct, non-overlapping zones for each skill.
The Dual Perspective: Exploration vs. Predictability
We can rewrite the exact same mutual information objective by expanding it from the perspective of state entropy:
$$I(S; Z) = H(S) - H(S|Z)$$
This decomposition reveals a beautiful mathematical tension that explains why mutual information produces structured control:
- Maximize $H(S)$ (State Entropy): The agent is incentivized to maximize the overall entropy of all states it visits across all skills. This acts as a powerful exploration drive, pushing the agent to expand outward and uncover the boundaries of its environment.
- Minimize $H(S|Z)$ (Conditional State Entropy): For any specific skill code $z$, the entropy of the visited states must be as low as possible. This means that once a skill code is selected, the agent's trajectory must be highly consistent, reliable, and predictable.
Optimizing both terms simultaneously prevents the agent from cheating. If the agent only maximized $H(S)$, it would act completely randomly, failing to learn reusable control primitives. If it only minimized $H(S|Z)$, it would stand completely still to ensure perfect predictability. Mutual information forces the agent to discover a diverse array of wide-ranging trajectories ($H(S)$) where each individual path is highly disciplined and reproducible ($H(S|Z)$).
As demonstrated by Hao Liu et al. in APS: Active Pretraining with Successor Features and Michael Laskin et al. in CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery, explicitly decoupling these two components allows agents to scale to complex, continuous control tasks.
Geometrically, this objective acts as a facility location problem. In The Information Geometry of Unsupervised Reinforcement Learning, Benjamin Eysenbach et al. prove that maximizing mutual information pushes the state distributions of different skills toward the outer vertices of the reachable state space, providing an optimal initialization for rapidly adapting to downstream tasks.
Refining the Objective: Dynamics and Relativity
While maximizing $I(S; Z)$ works well in simple environments, it has a notable vulnerability: an agent can satisfy the discriminator by simply finding static, distinct locations and "parking" there. To discover true skills—meaning coordinated patterns of motion—researchers shift the mutual information objective from static states to transitions and trajectories.
Dynamics-Aware Skills
In Dynamics-Aware Unsupervised Discovery of Skills, Archit Sharma et al. propose maximizing the mutual information between the next state $S_{t+1}$ and the skill $Z$, conditioned on the current state $S_t$:
$$I(S_{t+1}; Z | S_t) = H(S_{t+1}|S_t) - H(S_{t+1}|S_t, Z)$$
By conditioning on the current state, the objective ignores where the agent currently is and focuses entirely on how the skill changes the state. This forces the agent to learn predictable forward dynamics, allowing the discovered skills to be used directly by model-based planners for zero-shot downstream execution.
Relative Displacements
To completely decouple skills from absolute spatial coordinates, Relative Variational Intrinsic Control by Kate Baumli et al. maximizes the mutual information between the skill and the initial state, conditioned on the final state. This ensures that a skill is defined by the specific displacement or relative action performed, making the skills highly generalizable across different starting contexts.
Preventing Mode Collapse Across Paradigms
The utility of mutual information maximization extends far beyond pure unsupervised exploration. It serves as a fundamental mathematical tool to prevent mode collapse—the tendency of learning systems to discard behavioral diversity in favor of a single average solution.
Uncovering Multimodal Solutions in Offline RL
When training on static, offline datasets, standard reinforcement learning algorithms often suffer from averaging effects when the data contains multiple valid ways to solve a task. In Discovering Multiple Solutions from a Single Task in Offline Reinforcement Learning, Takayuki Osa et al. show that applying mutual information regularization between latent variables and state-action pairs forces the policy to remain highly sensitive to its latent inputs. This prevents the sub-policies from collapsing into a single mode, successfully extracting distinct, high-quality strategies from unlabelled data.
Spontaneous Specialization in Multi-Agent Systems
In multi-agent reinforcement learning, researchers frequently use parameter sharing (forcing all agents to use the same neural network) to improve training efficiency. However, this often causes agents to become identical clones, destroying the behavioral diversity needed for complex coordination.
Jiechuan Jiang addresses this in The Emergence of Individuality by maximizing the mutual information between an agent's unique identity vector and its trajectory of observations. By rewarding agents for behaving in ways that make them easily identifiable by a classifier, the system naturally triggers a division of labor and spontaneous role specialization without any manual reward engineering.
Key Foundational Work
Stochastic Neural Networks for Hierarchical Reinforcement Learning (2017) — contributed 2 concept elements to this direction.
Variational Intrinsic Control (2017) — contributed 1 concept elements to this direction.