Introduction
Unsupervised reinforcement learning (RL) aims to empower agents to discover a wide array of behaviors, or "skills," without the need for a manually engineered reward signal. While this paradigm excels at encouraging exploration, it often results in a "jack of all trades, master of none" agent. Without guidance, an agent may exhaustively explore behaviors that are mathematically diverse but practically useless, or even dangerous. For instance, a robot tasked with exploring a kitchen might learn dozens of ways to manipulate a knife, including several that pose a risk to itself or others, simply because those actions are novel.
To bridge the gap between autonomous exploration and human utility, we present Controlled Diversity with Preference (CDP). This framework leverages human feedback to steer an agent’s curiosity toward "preferred regions" of the environment, ensuring that the discovered skills are both diverse and aligned with human desires.
The Challenge of Unconstrained Discovery
Existing unsupervised RL methods typically rely on maximizing an information-theoretic objective, such as the mutual information between states and latent skill variables. While effective for coverage, these methods lack a mechanism to distinguish between "good" diversity and "bad" diversity. Traditional solutions to this problem—such as manual feature engineering or collecting expert demonstrations—are often too tedious or expensive to scale. CDP addresses this by introducing a human-in-the-loop component that provides high-level guidance through simple preferences.
The CDP Framework
The CDP approach builds upon the Exploration, Discovery, and Learning (EDL) framework, integrating human intuition into the skill discovery loop through four primary stages:
1. Reward Learning from Preferences
Instead of requiring a dense reward function, CDP utilizes a preference-based approach. A human supervisor is presented with pairs of short trajectory clips and asked to identify which behavior is more desirable. A reward model ($\hat{r}_\psi$) is then trained on these binary labels using the Bradley-Terry model, effectively capturing the human's underlying intent.
2. Defining Preferred Regions
Once the reward model is trained, the agent identifies "preferred regions" ($\hat{S}$) within the state space. These are defined as states where the predicted reward exceeds a user-defined threshold ($\beta$). This threshold acts as a control knob: a lower $\beta$ allows for broader, more diverse exploration, while a higher $\beta$ restricts the agent to behaviors that strictly align with the highest human preferences.
3. Guided Exploration
With the preferred regions defined, the agent employs a modified version of State Marginal Matching (SMM). The intrinsic reward is adjusted to prioritize states that satisfy three criteria: they must have high human preference, they must have low prior visit frequency (to encourage novelty), and they must be associated with a specific latent skill.
4. Preferred Latent Representation
A significant technical contribution of CDP is the use of the reward model's internal features as a "prior" for the skill discriminator. By using the hidden layers of the human-trained model, the agent ensures that the discovered skills are distinguishable based on features humans actually care about—such as velocity or posture—rather than arbitrary raw data points.
Experimental Evaluation
The authors evaluated CDP across two distinct simulated environments to test its ability to focus exploration and acquire meaningful skills.
- 2D Navigation: In a continuous 2D environment, CDP was tasked with discovering skills near a specific goal area. Compared to standard SMM and SMM with simple priors, CDP visited states with significantly higher rewards, demonstrating its ability to concentrate exploration within preferred manifolds.
- MuJoCo (Modified Half Cheetah): In this robotics benchmark, the desired behavior was defined as moving backward. While standard unsupervised methods failed to learn any relevant skills for this specific direction, CDP successfully discovered a variety of backward-moving skills at different speeds and gaits, all without an explicit velocity reward.
Key Insights
- Human Intuition as a Filter: CDP effectively uses human feedback to ignore irrelevant or "boring" parts of the environment, focusing the agent's limited computational budget on useful diversity.
- Feature Alignment: The "Preferred Latent Representation" ensures that the diversity the agent finds is "discernible" to human eyes, making the resulting skills more interpretable and applicable to downstream tasks.
- Query Efficiency: The iterative nature of the framework allows the agent to sample trajectories from high-preference regions and refine its model, making the process efficient enough for real-time human interaction.
Limitations and Future Work
While CDP marks a significant step toward safe and useful unsupervised RL, several challenges remain. The reliance on human labeling, though more efficient than providing demonstrations, still presents a bottleneck for long-horizon tasks. Additionally, the current framework assumes static human preferences; future iterations could explore how to adapt as human goals evolve during the exploration process. Finally, while CDP focuses on "desirability," explicitly incorporating hard safety constraints—actions the agent must never take—remains a critical area for future integration.
Summary updated