Unsupervised Curricula for Visual Meta-Reinforcement Learning

Allan Jabri, Kyle Hsu, Abhishek Gupta, Ben Eysenbach, Sergey Levine, Chelsea Finn

Meta-Reinforcement Learning (Meta-RL) is a powerful paradigm that teaches agents how to learn. Instead of mastering a single, isolated task, a meta-RL agent learns a strategy to quickly adapt to any new task drawn from a specific distribution. However, researchers usually have to hand-design these task distributions, which is tedious and often impossible in complex, high-dimensional visual environments (like a robot navigating via a camera feed).

Previous attempts at "unsupervised" meta-RL typically used a disjointed two-stage pipeline: first, discover skills, and then meta-learn on them. Unfortunately, existing skill-discovery methods often suffer from "mode collapse" when dealing with raw pixels, getting stuck repeating a few easy-to-distinguish behaviors rather than exploring the full breadth of the environment.

To address this, researchers introduce CARML (Continual Acquisition of RL Meta-Learning), an unsupervised algorithm that automatically generates a training curriculum for meta-RL agents in complex visual environments.

The CARML Approach: Closing the Loop

CARML frames unsupervised meta-RL as a variational Expectation-Maximization (EM) process, intertwining task discovery and meta-learning into a continuous loop.

The CARML algorithm alternates between organizing past experiences into skills and meta-training on those skills.

The algorithm operates in two main steps:

  1. The E-Step (Organize): The agent looks at its "reservoir" of past trajectories and fits a task scaffold—a mixture model that clusters experiences into discrete "skills." To handle raw pixels without mode collapse, CARML uses a technique inspired by DeepCluster, alternating between clustering representations and training a neural network to predict those cluster assignments.
  2. The M-Step (Meta-Train): The agent treats these clusters as its training tasks. It uses a meta-RL algorithm to learn a policy that can quickly identify which "skill" it is supposed to perform based on its initial observations and rewards.

The "reward" for these self-generated tasks uses a clever information-theoretic objective: $$r_z(s) = \lambda \log q_\phi(s|z) - \log q_\phi(s)$$ The first term rewards the agent for being consistent with a specific skill, while the second term acts as an exploration bonus, rewarding the agent for visiting states that are rare in its overall experience. The $\lambda$ parameter allows researchers to tune the balance between mastering known skills and exploring to find new ones.

During unsupervised pre-training, the policy meta-learns self-generated tasks. During transfer, it uses these acquired strategies to maximize unseen reward functions.

Key Insights and Results

The core "aha moment" of CARML is that task discovery and meta-learning are symbiotic. Meta-learning provides the agent with the ability to explore more effectively, which in turn leads to the discovery of more complex and structured tasks for the next round of meta-learning. Because the E-step periodically re-clusters the data, the "tasks" naturally evolve from simple movements to complex, object-oriented behaviors, creating an automatic curriculum.

The researchers evaluated CARML in challenging vision-based domains, including a first-person 3D ViZDoom navigation environment and a simulated Sawyer robotic arm manipulation task.

  • Zero-Shot Transfer: In ViZDoom, CARML agents achieved up to 71% success on human-defined navigation tasks without ever seeing a human reward function during training.
  • Sample Efficiency: Compared to standard RL (PPO) or curiosity-driven exploration (RND), CARML required two orders of magnitude fewer samples to solve downstream tasks.
  • Meta-Pretraining: CARML serves as an excellent pre-training step. When the agent was later moved to a supervised meta-RL setting, it reached peak performance much faster than an agent starting from scratch, proving that the unsupervised skills were structurally relevant to real-world tasks.

Limitations and Future Directions

While CARML represents a significant step forward, the authors note a "no-free-lunch" issue: unsupervised tasks do not always perfectly align with human goals. For example, in the Sawyer domain, the agent sometimes clustered tasks based on the robot arm's position rather than the object's position, simply because the arm is more visually prominent. Better inductive biases are needed to force the agent to focus on objects.

Additionally, the current model uses a fixed number of clusters for its skills. Future work could explore non-parametric models that allow the number of discovered skills to grow organically as the agent explores more of its world.

Summary updated

Abstract

In principle, meta-reinforcement learning algorithms leverage experience across many tasks to learn fast reinforcement learning (RL) strategies that transfer to similar tasks. However, current meta-RL approaches rely on manually-defined distributions of training tasks, and hand-crafting these task distributions can be challenging and time-consuming. Can "useful" pre-training tasks be discovered in an unsupervised manner? We develop an unsupervised algorithm for inducing an adaptive meta-training task distribution, i.e. an automatic curriculum, by modeling unsupervised interaction in a visual environment. The task distribution is scaffolded by a parametric density model of the meta-learner's trajectory distribution. We formulate unsupervised meta-RL as information maximization between a latent task variable and the meta-learner's data distribution, and describe a practical instantiation which alternates between integration of recent experience into the task distribution and meta-learning of the updated tasks. Repeating this procedure leads to iterative reorganization such that the curriculum adapts as the meta-learner's data distribution shifts. In particular, we show how discriminative clustering for visual representation can support trajectory-level task acquisition and exploration in domains with pixel observations, avoiding pitfalls of alternatives. In experiments on vision-based navigation and manipulation domains, we show that the algorithm allows for unsupervised meta-learning that transfers to downstream tasks specified by hand-crafted reward functions and serves as pre-training for more efficient supervised meta-learning of test task distributions.
BibTeX
@inproceedings{jabri2019unsupervised,
  author = {Allan Jabri and Kyle Hsu and Abhishek Gupta and Ben Eysenbach and Sergey Levine and Chelsea Finn},
  title = {{Unsupervised Curricula for Visual Meta-Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}

Loading related content...

Figure