Abstract
Choreographer is a model-based reinforcement learning (RL) agent designed to learn a diverse repertoire of behaviors entirely within a "world model." By decoupling exploration from skill discovery and utilizing a meta-controller to evaluate skills in "imagination," Choreographer addresses the limitations of current unsupervised RL methods. It achieves state-of-the-art performance and high data efficiency on standard benchmarks by identifying optimal behaviors without extensive real-world interaction.
Introduction
Unsupervised Reinforcement Learning (URL) aims to develop agents capable of exploring environments and acquiring useful skills without external rewards. Ideally, such agents should adapt rapidly to specific tasks once they are introduced. However, existing competence-based methods often suffer from two major hurdles: limited exploration—where agents get trapped in previously visited states—and inefficient adaptation, which typically requires costly real-world trials to evaluate which learned skills are effective for a new task.
Methodology
Choreographer leverages a World Model, a generative system that learns the physics and transition rules of an environment. This allows the agent to "imagine" outcomes rather than relying solely on physical experience.
Skill Discovery and Learning
The agent employs a Vector Quantized Autoencoder (VQ-VAE) to partition the latent states of the world model into a discrete "codebook" of skills. To ensure a diverse library of behaviors, a code resampling technique is used to prevent "index collapse," where only a subset of skills is utilized.
Skills are trained entirely within the world model. The reward function for these policies, $r_{skill}$, approximates the maximization of mutual information $I(\tau_s, z)$ between state trajectories and skill codes. As shown in the methodology diagram, this is achieved by combining a reward for reaching the state associated with a specific code ($r_{code}$) with a reward for maximizing state entropy ($r_{entropy}$), encouraging dynamic and diverse movement.
Training Phases
The Choreographer framework operates in two distinct phases: an unsupervised phase for broad skill acquisition and a supervised phase for task-specific adaptation.
- Unsupervised Phase: The agent collects data through exploration and pre-trains its model and behaviors using unlabelled data. This phase focuses on building a robust library of stable behaviors.
- Supervised Phase (Fine-tuning): When a task reward is introduced, an imagination-based meta-controller evaluates the learned skills by simulating their outcomes. This "hallucination" allows the agent to select the most effective skill for the new task before taking a single real-world step, significantly improving data efficiency.
Results and Discussion
Choreographer was evaluated on the URL Benchmark (URLB), ExORL, and Meta-World domains.
- Performance: It outperformed strong baselines such as CIC, ProtoRL, and APT in both state-based and pixel-based settings.
- Sparse Rewards: In complex robotic tasks like the Jaco arm reaching, Choreographer achieved a 100% success rate, whereas most baselines failed to find the target even once.
- Efficiency: The ability to learn and evaluate in imagination allowed for near-instant adaptation compared to traditional trial-and-error methods.
Conclusion
Choreographer demonstrates that decoupling data collection from skill discovery and utilizing a world model for "imagined" adaptation provides a robust path forward for URL. While the agent's performance is dependent on the accuracy of its world model, the approach offers a powerful framework for building versatile robotic agents. Future research will focus on hierarchical sequencing of skills to solve long-horizon, multi-stage tasks.
Summary updated