TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations

Junik Bae, Kwanyoung Park, Youngwoon Lee

TL;DR

This paper introduces TLDR (TemporaL Distance-aware Representations), a novel unsupervised goal-conditioned reinforcement learning method that uses a learned metric of "shortest temporal distance" to unify exploration and goal-reaching. By selecting goals that are temporally farthest from visited states and optimizing policies to minimize these step-based distances, TLDR achieves vastly superior state coverage in complex, long-horizon environments compared to prior state-of-the-art methods.

What Problem Does This Paper Address?

Unsupervised Goal-Conditioned Reinforcement Learning (GCRL) aims to develop autonomous agents capable of reaching any arbitrary goal state without relying on hand-crafted external reward functions. This serves as a powerful paradigm for robotic pre-training.

However, existing unsupervised GCRL methods face two major bottlenecks:

  1. Ineffective Exploration: Standard exploration strategies focus on finding "novel" states or states with high dynamics uncertainty. In complex environments (like intricate mazes), this often fails to guide the agent toward truly meaningful, hard-to-reach areas, causing them to get stuck in localized regions.
  2. Sparse or Heuristic Rewards: Training a policy to reach a goal usually relies on sparse rewards (did the agent hit the exact coordinate?) or brittle heuristics (like cosine similarity), which perform poorly over long horizons.

Methods

The core idea behind TLDR is to leverage temporal distance—the minimum number of environment steps required to transition between two states—as a dense, task-agnostic metric for every stage of the learning process.

Figure 2: Overview of TLDR algorithm. TLDR leverages temporal distance-aware representations for unsupervised GCRL. (a) We start by learning a state encoder ϕ(s) that maps states to temporal distance-aware representations. With the temporal distance-aware representations, TLDR (b) selects the temporally farthest state from the visited states as an exploratory goal, (c) reaches the chosen goal using a goal-conditioned policy, which learns to minimize temporal distance to the goal, and (d) collects exploratory trajectories using an exploration policy that visits states with large temporal distance from the visited states.

TLDR builds upon the classic "Go-Explore" framework through four key components:

  • Temporal Distance-Aware Representations: The authors train a state encoder $\phi(s)$ using a constrained optimization objective (inspired by Quasimetric RL). This maps states into a latent space where the Euclidean distance $\|\phi(s_1) - \phi(s_2)\|$ directly approximates the shortest temporal distance between them.
  • Exploratory Goal Selection: At the start of an episode, TLDR uses a particle-based entropy estimator on top of the latent space to sample a goal from the replay buffer that is temporally farthest away from already visited states.
  • Goal-Conditioned Policy (Go-phase): The agent learns to navigate to the selected goal by maximizing a dense reward defined by the decrease in temporal distance to the goal: $$r^G(s, s', g) = \|\phi(s) - \phi(g)\| - \|\phi(s') - \phi(g)\|$$
  • Exploration Policy (Explore-phase): Once the agent gets close to the goal, an explicit exploration policy takes over to gather new experiences by maximizing the temporal distance even further away from the visited state distribution.

Datasets

The authors evaluated TLDR across a diverse suite of continuous control and locomotion environments:

  • State-based Locomotion: Ant and HalfCheetah (OpenAI Gym), Humanoid-Run and Quadruped-Escape (DeepMind Control Suite).
  • Complex Mazes: AntMaze-Large (D4RL) and AntMaze-Ultra, which require navigating highly intricate, long-horizon layouts.
  • Pixel-based Environments: Quadruped (Pixel) and Kitchen (Pixel) (D4RL) to test performance directly from raw $64 \times 64 \times 3$ image observations.
  • Real-world Proxy: A simulated Unitree A1 quadruped robot (12 Degrees of Freedom) to evaluate potential sim-to-real applicability.

Key Insights and Ideas

  • Shortest vs. Expected Distance: Prior methods like LEXA and PEG use expected temporal distances based on the current policy. TLDR instead estimates the shortest possible temporal distance across all collected data. This provides a much more stable and grounded directional gradient for the goal-conditioned policy, preventing circuitous, sub-optimal paths.
  • Density vs. Reachability: Traditional novelty-based exploration looks for low-density states. However, a state being low-density doesn't mean it's useful or hard to reach. By framing exploration around maximizing temporal distance, TLDR explicitly forces the agent to push the boundaries of its reachable universe.
  • Compatibility with Safety Constraints: Because temporal distance is an intrinsic metric, it can easily be combined with external regularizers. For instance, adding a penalty for falling over allowed a simulated quadruped to safely maximize its exploration without degrading its state coverage.

Key Results

  • Unprecedented Maze Coverage: In the challenging AntMaze-Large and AntMaze-Ultra environments, prior state-of-the-art methods like METRA and PEG failed to escape localized areas near the spawn point. TLDR successfully explored virtually the entire maze layout.
Figure 1: Trajectories (red) of an ant robot in a complex maze trained by TLDR, METRA [1], and PEG [2]. While prior methods yield limited exploration, TLDR explores the entire maze.
  • Quantitative Goal Reaching: TLDR significantly outperformed baselines in goal-directed evaluation. In AntMaze-Ultra, TLDR successfully reached almost all 21 pre-defined evaluation goals, whereas prior methods hovered near zero.
  • Simulated Quadruped Success: On the Unitree A1 robot simulation, TLDR achieved substantially higher state coverage and lower final goal distances than METRA within a fixed 6-hour training window.

Limitations and Future Work

Despite its strong performance, the authors highlight several open challenges for TLDR:

  • The Pixel Bottleneck: While highly effective on state-based inputs, TLDR learns slowly when trained purely on pixel observations. Component-wise analysis revealed that learning robust goal-conditioned representations from raw images remains a major bottleneck.
  • Assumption of Symmetry: The Euclidean distance metric used in the latent space assumes that the distance from state $A$ to $B$ is identical to $B$ to $A$. This assumption breaks down in highly asymmetric environments (e.g., environments with strong currents, steep drops, or one-way doors).
  • Sample Efficiency: While TLDR is highly efficient in terms of wall-clock time, it requires a large number of environment steps. Integrating model-based RL approaches could drastically improve its sample efficiency.

Summary updated

Abstract

Unsupervised goal-conditioned reinforcement learning (GCRL) is a promising paradigm for developing diverse robotic skills without external supervision. However, existing unsupervised GCRL methods often struggle to cover a wide range of states in complex environments due to their limited exploration and sparse or noisy rewards for GCRL. To overcome these challenges, we propose a novel unsupervised GCRL method that leverages TemporaL Distance-aware Representations (TLDR). Based on temporal distance, TLDR selects faraway goals to initiate exploration and computes intrinsic exploration rewards and goal-reaching rewards. Specifically, our exploration policy seeks states with large temporal distances (i.e. covering a large state space), while the goal-conditioned policy learns to minimize the temporal distance to the goal (i.e. reaching the goal). Our results in six simulated locomotion environments demonstrate that TLDR significantly outperforms prior unsupervised GCRL methods in achieving a wide range of states.
BibTeX
@inproceedings{bae2024tldr,
  author = {Junik Bae and Kwanyoung Park and Youngwoon Lee},
  title = {{TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations}},
  booktitle = {Conference on Robot Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=Nb7RtUVbeQ},
}
Figure