Abstract
Unsupervised goal-conditioned reinforcement learning (GCRL) is a promising paradigm for developing diverse robotic skills without external supervision. However, existing unsupervised GCRL methods often struggle to cover a wide range of states in complex environments due to their limited exploration and sparse or noisy rewards for GCRL. To overcome these challenges, we propose a novel unsupervised GCRL method that leverages TemporaL Distance-aware Representations (TLDR). TLDR selects faraway goals to initiate exploration and computes intrinsic exploration rewards and goal-reaching rewards, based on temporal distance. Specifically, our exploration policy seeks states with large temporal distances (i.e. covering a large state space), while the goal-conditioned policy learns to minimize the temporal distance to the goal (i.e. reaching the goal). Our experimental results in six simulated robotic locomotion environments demonstrate that our method significantly outperforms previous unsupervised GCRL methods in achieving a wide variety of states.
Introduction
In the quest to build "generalist" robots, researchers use Unsupervised Goal-Conditioned Reinforcement Learning (GCRL). The ultimate goal is an agent that, like a human infant, interacts with its environment to learn what is possible before being assigned a specific task. However, current methods often fail in complex environments like large mazes due to poor exploration—where agents get stuck in nearby areas—and sparse rewards, where binary feedback makes reaching distant goals difficult. Standard metrics like Euclidean distance often fail when physical obstacles make straight-line paths impossible.
TLDR (TemporaL Distance-aware Representations) addresses these issues by using "temporal distance"—the number of steps required to travel between states—as a universal compass for robotic navigation.
Methodology
TLDR integrates temporal distance into every stage of the learning process to enable autonomous exploration and skill acquisition without human supervision.
- Representation Learning: The agent learns an encoder that maps raw states into a latent space where the Euclidean distance between points represents the minimum number of environment steps needed to travel between them.
- Goal Selection: Instead of random sampling, TLDR identifies states in its replay buffer that are "temporally distant" from previously visited states to serve as exploration targets.
- The Go-Explore Loop: The agent first uses a goal-conditioned policy to navigate to a chosen "faraway" state (Go-phase). Once reached, it switches to an exploration policy designed to move even further into unknown territory (Explore-phase).
- Dense Rewards: By leveraging the learned temporal map, the agent provides itself with a dense reward signal based on whether it is moving temporally closer to its goal, smoothing the learning of long-horizon tasks.
Experimental Results
The researchers evaluated TLDR across challenging robotic simulation benchmarks, including AntMaze-Large/Ultra, Humanoid-Run, and pixel-based environments like Kitchen.
Key findings include:
- Superior Coverage: In the "AntMaze-Ultra" environment, TLDR explored almost the entire maze, while prior methods like METRA and PEG remained confined to the initial corridors.
- Zero-Shot Success: After unsupervised pre-training, TLDR successfully reached 21 different pre-defined goals in the Ultra maze, a task where previous methods achieved near-zero success.
- Robustness: TLDR consistently outperformed six major unsupervised RL algorithms across nearly all tested environments in terms of state coverage.
- Safety Integration: The framework can be combined with safety rewards (e.g., keeping a robot upright), making it potentially viable for real-world hardware.
Discussion and Future Work
TLDR demonstrates that temporal distance is often a superior metric to spatial distance in complex environments, as it accounts for obstacles that spatial metrics ignore. By maximizing temporal distance for exploration and minimizing it for goal-reaching, TLDR unifies two traditionally separate RL problems.
However, limitations remain. TLDR requires a large number of environment interactions to learn its representations, and learning directly from raw pixels is significantly more difficult than from state vectors. Additionally, the current model assumes symmetric travel times (A to B is the same as B to A), which is not always true in the real world. Future work will focus on visual representation learning and handling asymmetric temporal distances.
Summary updated