Temporal abstractions-augmented temporally contrastive learning: an alternative to the Laplacian in RL

Akram Erraqabi, Marlos C. Machado, Harry Zhao, Sainbayar Sukhbaatar, Ludovic Denoyer, Alessandro Lazaric, Yoshua Bengio

Introduction

Researchers have developed TATC (Temporal Abstractions-Augmented Temporally Contrastive Learning), a new method for learning state representations in reinforcement learning (RL) without needing a "god-mode" reset to every state. By combining representation learning with a "boredom-driven" exploration policy, TATC recovers the powerful properties of the graph Laplacian—traditionally used for skill discovery and reward shaping—even in challenging environments where the agent starts from a single fixed point.

The "Chicken-and-Egg" Problem of Exploration

In task-agnostic RL, we want an agent to explore an environment and build a "map" (a representation) of how states relate to one another before it even knows what its specific goal is. A popular tool for this is the graph Laplacian, which captures the underlying geometry and dynamics of the environment.

However, current state-of-the-art methods for approximating the Laplacian (like LAP-REP) rely on a "uniform prior." This means they assume the agent can magically teleport to any state in the environment to collect data. In reality, agents usually start at a fixed point and must explore outward. Without uniform access, standard Laplacian methods fail because they only learn about the small area near the starting point. This creates a "chicken-and-egg" problem: you need a good representation to explore, but you need to explore to get a good representation.

Methods: Breaking the Cycle with TATC

The authors introduce TATC, which breaks this cycle by tying representation learning to a hierarchical exploration strategy. Key components include:

  1. Hierarchical Skills: The agent uses a bi-level policy. A high-level policy picks a "direction" in the current representation space, and a low-level policy learns "skills" to move the agent in that direction.
  2. Temporally-Contrastive Objective: Like previous work, TATC uses a loss function that pulls temporally close states together and pushes far-apart states away in the representation space.
  3. The "Boredom" Term: This is the secret sauce. The authors add a "contracting" term to the loss function that brings states connected by a learned skill closer together. As the agent repeats a skill, the representation of that path "shrinks," making the high-level reward for that direction smaller. This effectively makes the agent "bored" with known paths, forcing it to pick new directions and push the frontier of the explored area.
  4. A Virtuous Cycle: As the representation improves, the skills become more precise. As the skills allow for deeper exploration, the agent collects more diverse data to further refine the representation.
The TATC virtuous cycle of representation learning and skill discovery

Experimental Setup

The method was evaluated on two main types of environments:

  • Gridworlds: Discrete environments including U-Maze, T-Maze, and 4-Rooms. These are classic benchmarks for testing if a representation can respect walls and navigate intersections.
  • Continuous Control (MuJoCo): More complex, high-dimensional environments using the Ant agent in two large mazes (AntMaze-1 and AntMaze-2). These require the agent to coordinate multiple joints to move through a continuous space.

Key Insights and Results

  • Recovering the Laplacian: TATC successfully recovers the expressive power of the Laplacian representation in "non-uniform" settings where previous methods completely break down.
Evolution of the learned representation over time, showing how it unfolds to capture the environment's geometry
  • Dynamics-Awareness: By using skills to define the representation, the "distance" in the learned space actually reflects how hard it is to travel between points, rather than just how close they look. For example, two states on opposite sides of a thin wall are "far" in TATC but "close" in Euclidean space.
  • Emergent Exploration: Exploration isn't just a random walk; it's driven by the representation's own "boredom." This allows the agent to solve sparse-reward tasks (where the agent only gets a reward at the very end) much faster than standard skill-discovery methods like DIAYN or DCO.
  • Value Prediction and Reward Shaping: In gridworlds, TATC's representations allowed a linear critic to approximate the optimal value function as accurately as if the agent had been trained with a uniform prior. In the MuJoCo Ant Mazes, using TATC representations to "shape" the reward allowed the agent to solve the maze, while LAP-REP failed to guide the agent to the goal.
  • Sparse Rewards: In a "blind" navigation task with no dense reward, TATC-trained skills reached the goal in ~5 epochs, whereas competing methods failed to solve the task within 50 epochs.

Limitations and Future Work

The authors note that while TATC is powerful, it introduces new hyperparameters, such as the strength of the boredom term, which may require tuning. Additionally, while the learned skills are excellent for exploration, they aren't explicitly optimized for "diversity" in the same way some other frameworks are. Future work could involve scaling this to even more complex visual environments or integrating it with model-based RL, where the representation could serve as the foundation for a latent world model.

Summary updated

Abstract

In reinforcement learning (RL), the graph Laplacian has proved to be a valuable tool in the task-agnostic setting, with applications ranging from option discovery to dynamics-aware metric learning. Conveniently, learning the Laplacian representation has recently been framed as the optimization of a temporally-contrastive objective to overcome its computational limitations in large or even continuous state spaces (Wu et al., 2019). However, this approach relies on a uniform access to the state space S, and overlooks the exploration problem that emerges during the representation learning process. In this work, we reconcile such representation learning with exploration in a non-uniform prior setting, while recovering the expressive potential afforded by a uniform prior. Our approach leverages the learned representation to build a skill-based covering policy which in turn provides a better training distribution to extend and refine the representation. We also propose to integrate temporal abstractions captured by the learned skills into the representation, which encourages exploration and improves the representation’s dynamics-awareness. We find that our method scales better to challenging environments, and that the learned skills can solve difficult continuous navigation tasks with sparse rewards, where standard skill discovery methods are limited.
BibTeX
@inproceedings{erraqabi2022temporal,
  author = {Akram Erraqabi and Marlos C. Machado and Harry Zhao and Sainbayar Sukhbaatar and Ludovic Denoyer and Alessandro Lazaric and Yoshua Bengio},
  title = {{Temporal abstractions-augmented temporally contrastive learning: an alternative to the Laplacian in RL}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=bUKyC0UiZcr},
}
Figure