Choreographer: Learning and Adapting Skills in Imagination

Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Alexandre Lacoste, Sai Rajeswar

Abstract

Choreographer is a model-based reinforcement learning (RL) agent designed to learn a diverse repertoire of behaviors entirely within a "world model." By decoupling exploration from skill discovery and utilizing a meta-controller to evaluate skills in "imagination," Choreographer addresses the limitations of current unsupervised RL methods. It achieves state-of-the-art performance and high data efficiency on standard benchmarks by identifying optimal behaviors without extensive real-world interaction.

Introduction

Unsupervised Reinforcement Learning (URL) aims to develop agents capable of exploring environments and acquiring useful skills without external rewards. Ideally, such agents should adapt rapidly to specific tasks once they are introduced. However, existing competence-based methods often suffer from two major hurdles: limited exploration—where agents get trapped in previously visited states—and inefficient adaptation, which typically requires costly real-world trials to evaluate which learned skills are effective for a new task.

Methodology

Choreographer leverages a World Model, a generative system that learns the physics and transition rules of an environment. This allows the agent to "imagine" outcomes rather than relying solely on physical experience.

Skill Discovery and Learning

The agent employs a Vector Quantized Autoencoder (VQ-VAE) to partition the latent states of the world model into a discrete "codebook" of skills. To ensure a diverse library of behaviors, a code resampling technique is used to prevent "index collapse," where only a subset of skills is utilized.

The skill discovery process where a skill policy is conditioned on a latent code $z$ to maximize mutual information through state entropy and code reaching.

Skills are trained entirely within the world model. The reward function for these policies, $r_{skill}$, approximates the maximization of mutual information $I(\tau_s, z)$ between state trajectories and skill codes. As shown in the methodology diagram, this is achieved by combining a reward for reaching the state associated with a specific code ($r_{code}$) with a reward for maximizing state entropy ($r_{entropy}$), encouraging dynamic and diverse movement.

Training Phases

The Choreographer framework operates in two distinct phases: an unsupervised phase for broad skill acquisition and a supervised phase for task-specific adaptation.

Overview of the Choreographer framework, showing the transition from unsupervised data collection and pre-training to supervised fine-tuning.
  1. Unsupervised Phase: The agent collects data through exploration and pre-trains its model and behaviors using unlabelled data. This phase focuses on building a robust library of stable behaviors.
  2. Supervised Phase (Fine-tuning): When a task reward is introduced, an imagination-based meta-controller evaluates the learned skills by simulating their outcomes. This "hallucination" allows the agent to select the most effective skill for the new task before taking a single real-world step, significantly improving data efficiency.

Results and Discussion

Choreographer was evaluated on the URL Benchmark (URLB), ExORL, and Meta-World domains.

  • Performance: It outperformed strong baselines such as CIC, ProtoRL, and APT in both state-based and pixel-based settings.
  • Sparse Rewards: In complex robotic tasks like the Jaco arm reaching, Choreographer achieved a 100% success rate, whereas most baselines failed to find the target even once.
  • Efficiency: The ability to learn and evaluate in imagination allowed for near-instant adaptation compared to traditional trial-and-error methods.

Conclusion

Choreographer demonstrates that decoupling data collection from skill discovery and utilizing a world model for "imagined" adaptation provides a robust path forward for URL. While the agent's performance is dependent on the accuracy of its world model, the approach offers a powerful framework for building versatile robotic agents. Future research will focus on hierarchical sequencing of skills to solve long-horizon, multi-stage tasks.

Summary updated

BibTeX
@inproceedings{mazzaglia2023choreographer,
  author = {Pietro Mazzaglia and Tim Verbelen and Bart Dhoedt and Alexandre Lacoste and Sai Rajeswar},
  title = {{Choreographer: Learning and Adapting Skills in Imagination}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=PhkWyijGi5b},
}
Figure