TL;DR
METRA is a novel unsupervised reinforcement learning (RL) framework that learns diverse, useful behaviors by maximizing coverage in a compact latent space metrically tied to the environment via temporal distances. It is the first unsupervised method to successfully discover diverse locomotion skills (like running in different directions) in complex, pixel-based environments such as the Quadruped and Humanoid.
What Problem Does This Paper Address?
Unsupervised RL aims to train agents to discover a wide range of useful behaviors without any external reward signal. Ideally, these "pre-trained" agents can then be quickly adapted to specific downstream tasks. However, existing approaches face a scalability wall:
- Pure Exploration methods (like RND or APT) try to visit every possible state. In high-dimensional spaces, they often get "stuck" exploring irrelevant details—like a robot obsessively wiggling its toes instead of learning to walk.
- Mutual Information (MI) Skill Discovery (like DIAYN or DADS) tries to learn "distinguishable" skills. Because MI is based on KL-divergence, it is "metric-agnostic." It doesn't care how far apart behaviors are, only that they are different. This often results in the agent learning many tiny, static, and ultimately useless poses.
The challenge is to create an objective that encourages agents to learn behaviors that are not just different, but metrically distant and meaningful, especially in pixel-based environments where simple Euclidean distance between images is useless.
Methods
The authors propose Metric-Aware Abstraction (METRA). The core idea is to learn a mapping from the high-dimensional state space $S$ to a compact latent space $Z$ such that distances in $Z$ reflect temporal distances in the environment (i.e., the minimum number of steps to get from state A to state B).
Key Innovations:
- Wasserstein Dependency Measure (WDM): Instead of the standard Mutual Information objective, METRA uses a Wasserstein-based objective. Unlike MI, WDM is metric-aware; it explicitly rewards the agent for reaching states that are far away according to a chosen metric.
- Temporal Distance as a Metric: METRA uses the number of environment steps as its distance metric. This is a "natural" metric for RL because it is invariant to how the state is represented (pixels vs. joint angles) and respects the underlying physics of the world.
- Temporal PCA: The authors prove that under certain conditions, METRA is equivalent to performing Principal Component Analysis (PCA) in the temporal embedding space. It identifies the "principal components" of movement—the directions in which the agent can travel the furthest.
Datasets
The authors evaluated METRA on five challenging robotic locomotion and manipulation environments:
- State-based: MuJoCo Ant and HalfCheetah.
- Pixel-based (64x64 images): DeepMind Control Suite Quadruped and Humanoid.
- Pixel-based Manipulation: A version of the Franka Kitchen environment.
To allow the agent to infer its global position from pixels in the locomotion tasks, they used gradient-colored floors, a standard practice for pixel-based navigation benchmarks.
Key Insights and Ideas
- Metric Matters: The "aha moment" is realizing that distinguishability (MI) isn't enough for discovery. You need a sense of distance. By forcing skills to be temporally distant, the agent is incentivized to move as far and as fast as possible in various directions.
- Abstraction is Scalable: You don't need to cover the entire state space. By covering a compact, temporally-linked latent space, METRA ignores high-frequency "noise" (like specific leg poses) and focuses on "macro" behaviors (like running directions).
- Zero-Shot Capabilities: Because the learned representation $\phi(s)$ preserves temporal distances, the agent can reach new goals in a zero-shot manner. To reach goal $g$ from state $s$, the agent simply selects the skill $z$ that aligns with the vector $\phi(g) - \phi(s)$.
Key Results
- First of its Kind: METRA is the first unsupervised RL method to discover diverse locomotion skills (running in all directions) in pixel-based Quadruped and Humanoid environments.
- Superior Coverage: In quantitative tests, METRA significantly outperformed 11 state-of-the-art baselines (including LSD, CIC, and LBS) in terms of state space coverage.
- Downstream Performance: Skills learned by METRA proved more useful for downstream tasks. When training a high-level controller on top of the pre-trained skills, METRA achieved the highest rewards across all five benchmark environments.
Limitations and Future Work
- Symmetry Assumption: The current implementation embeds temporal distance into a symmetric Euclidean latent space. This works well for most locomotion but might struggle in "one-way" environments (e.g., falling down a cliff) where temporal distance is highly asymmetric.
- Linear Latent Dynamics: The simplified objective assumes behaviors move linearly in the latent space. While the mapping $\phi$ is non-linear, exploring more complex latent dynamics could unlock even more diverse behaviors.
- Sample Efficiency: METRA currently uses vanilla Soft Actor-Critic (SAC). The authors suggest that combining METRA with more sample-efficient model-based RL backbones is a promising future direction.
Summary updated