METRA: Scalable Unsupervised RL with Metric-Aware Abstraction

Seohong Park, Oleh Rybkin, Sergey Levine

TL;DR

METRA is a novel unsupervised reinforcement learning (RL) framework that learns diverse, useful behaviors by maximizing coverage in a compact latent space metrically tied to the environment via temporal distances. It is the first unsupervised method to successfully discover diverse locomotion skills (like running in different directions) in complex, pixel-based environments such as the Quadruped and Humanoid.

What Problem Does This Paper Address?

Unsupervised RL aims to train agents to discover a wide range of useful behaviors without any external reward signal. Ideally, these "pre-trained" agents can then be quickly adapted to specific downstream tasks. However, existing approaches face a scalability wall:

  1. Pure Exploration methods (like RND or APT) try to visit every possible state. In high-dimensional spaces, they often get "stuck" exploring irrelevant details—like a robot obsessively wiggling its toes instead of learning to walk.
  2. Mutual Information (MI) Skill Discovery (like DIAYN or DADS) tries to learn "distinguishable" skills. Because MI is based on KL-divergence, it is "metric-agnostic." It doesn't care how far apart behaviors are, only that they are different. This often results in the agent learning many tiny, static, and ultimately useless poses.

The challenge is to create an objective that encourages agents to learn behaviors that are not just different, but metrically distant and meaningful, especially in pixel-based environments where simple Euclidean distance between images is useless.

Methods

The authors propose Metric-Aware Abstraction (METRA). The core idea is to learn a mapping from the high-dimensional state space $S$ to a compact latent space $Z$ such that distances in $Z$ reflect temporal distances in the environment (i.e., the minimum number of steps to get from state A to state B).

Figure 1: Illustration of METRA. Our main idea for scalable unsupervised RL is to cover only the most “important” low-dimensional subset of the state space, analogously to PCA. Specifically, METRA covers the most “temporally spread-out” (non-linear) manifold, which would lead to approximate coverage of the state space S. In the example above, the two-dimensional Z space captures behaviors running in all directions, not necessarily covering every possible leg pose.

Key Innovations:

  • Wasserstein Dependency Measure (WDM): Instead of the standard Mutual Information objective, METRA uses a Wasserstein-based objective. Unlike MI, WDM is metric-aware; it explicitly rewards the agent for reaching states that are far away according to a chosen metric.
  • Temporal Distance as a Metric: METRA uses the number of environment steps as its distance metric. This is a "natural" metric for RL because it is invariant to how the state is represented (pixels vs. joint angles) and respects the underlying physics of the world.
  • Temporal PCA: The authors prove that under certain conditions, METRA is equivalent to performing Principal Component Analysis (PCA) in the temporal embedding space. It identifies the "principal components" of movement—the directions in which the agent can travel the furthest.

Datasets

The authors evaluated METRA on five challenging robotic locomotion and manipulation environments:

  • State-based: MuJoCo Ant and HalfCheetah.
  • Pixel-based (64x64 images): DeepMind Control Suite Quadruped and Humanoid.
  • Pixel-based Manipulation: A version of the Franka Kitchen environment.

To allow the agent to infer its global position from pixels in the locomotion tasks, they used gradient-colored floors, a standard practice for pixel-based navigation benchmarks.

Key Insights and Ideas

  • Metric Matters: The "aha moment" is realizing that distinguishability (MI) isn't enough for discovery. You need a sense of distance. By forcing skills to be temporally distant, the agent is incentivized to move as far and as fast as possible in various directions.
  • Abstraction is Scalable: You don't need to cover the entire state space. By covering a compact, temporally-linked latent space, METRA ignores high-frequency "noise" (like specific leg poses) and focuses on "macro" behaviors (like running directions).
  • Zero-Shot Capabilities: Because the learned representation $\phi(s)$ preserves temporal distances, the agent can reach new goals in a zero-shot manner. To reach goal $g$ from state $s$, the agent simply selects the skill $z$ that aligns with the vector $\phi(g) - \phi(s)$.
Figure 2: Sketch comparing different unsupervised RL objectives. Pure exploration approaches try to cover every possible state, which is infeasible in complex environments (e.g., such methods might be “stuck” at forever finding novel joint angle configurations of a robot, without fully exploring the environment; see Figure 3). The mutual information I(S; Z) has no underlying distance metrics, and thus does not prioritize coverage enough, only focusing on skills that are discriminable. In contrast, our proposed Wasserstein de- pendency measure IW(S; Z) maximizes the distance metric d, which we choose to be the temporal distance, forcing the learned skills to span the “longest” subspaces of the state space, analogously to (temporal, non- linear) PCA.

Key Results

  • First of its Kind: METRA is the first unsupervised RL method to discover diverse locomotion skills (running in all directions) in pixel-based Quadruped and Humanoid environments.
  • Superior Coverage: In quantitative tests, METRA significantly outperformed 11 state-of-the-art baselines (including LSD, CIC, and LBS) in terms of state space coverage.
  • Downstream Performance: Skills learned by METRA proved more useful for downstream tasks. When training a high-level controller on top of the pre-trained skills, METRA achieved the highest rewards across all five benchmark environments.
Figure 3: Examples of behaviors learned by 11 unsupervised RL methods. For locomotion environments, we plot the x-y (or x) trajectories sampled from learned policies. For Kitchen, we measure the coincidental success rates for six predefined tasks. Different colors represent different skills z. METRA is the only method that discovers diverse locomotion skills in pixel-based Quadruped and Humanoid. We refer to Figure 12 for the complete qualitative results (8 seeds) of METRA and our project page for videos.

Limitations and Future Work

  • Symmetry Assumption: The current implementation embeds temporal distance into a symmetric Euclidean latent space. This works well for most locomotion but might struggle in "one-way" environments (e.g., falling down a cliff) where temporal distance is highly asymmetric.
  • Linear Latent Dynamics: The simplified objective assumes behaviors move linearly in the latent space. While the mapping $\phi$ is non-linear, exploring more complex latent dynamics could unlock even more diverse behaviors.
  • Sample Efficiency: METRA currently uses vanilla Soft Actor-Critic (SAC). The authors suggest that combining METRA with more sample-efficient model-based RL backbones is a promising future direction.

Summary updated

Abstract

Unsupervised pre-training strategies have proven to be highly effective in natural language processing and computer vision. Likewise, unsupervised reinforcement learning (RL) holds the promise of discovering a variety of potentially useful behaviors that can accelerate the learning of a wide array of downstream tasks. Previous unsupervised RL approaches have mainly focused on pure exploration and mutual information skill learning. However, despite the previous attempts, making unsupervised RL truly scalable still remains a major open challenge: pure exploration approaches might struggle in complex environments with large state spaces, where covering every possible transition is infeasible, and mutual information skill learning approaches might completely fail to explore the environment due to the lack of incentives. To make unsupervised RL scalable to complex, high-dimensional environments, we propose a novel unsupervised RL objective, which we call Metric-Aware Abstraction (METRA). Our main idea is, instead of directly covering the entire state space, to only cover a compact latent space $Z$ that is metrically connected to the state space $S$ by temporal distances. By learning to move in every direction in the latent space, METRA obtains a tractable set of diverse behaviors that approximately cover the state space, being scalable to high-dimensional environments. Through our experiments in five locomotion and manipulation environments, we demonstrate that METRA can discover a variety of useful behaviors even in complex, pixel-based environments, being the first unsupervised RL method that discovers diverse locomotion behaviors in pixel-based Quadruped and Humanoid. Our code and videos are available at https://seohong.me/projects/metra/
BibTeX
@inproceedings{park2024metra,
  author = {Seohong Park and Oleh Rybkin and Sergey Levine},
  title = {{METRA: Scalable Unsupervised RL with Metric-Aware Abstraction}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=c5pwL0Soay},
}
Figure