Discrete Compositional Representations as an Abstraction for Goal Conditioned Reinforcement Learning

Riashat Islam, Hongyu Zang, Anirudh Goyal, Alex M. Lamb, Kenji Kawaguchi, Xin Li, Romain Laroche, Yoshua Bengio, Remi Tachet des CombesNeurIPS 2022 · OpenReview

In the world of Reinforcement Learning (RL), we often want agents that don't just do one thing, but can reach any goal we set for them. This is known as Goal-Conditioned RL (GCRL). However, teaching an agent to understand a goal from a high-dimensional image (like "pick up the red apple") is notoriously difficult.

A paper from researchers at McGill, Mila, Microsoft, and DeepMind, titled "Discrete Compositional Representations as an Abstraction for Goal Conditioned RL," proposes a clever solution: stop treating goals as continuous points in space and start treating them as a collection of discrete, "factorial" building blocks.

Conceptual overview of discrete factorial representations

TL;DR

The authors introduce DGRL, a method that uses a vector-quantization (VQ) bottleneck to turn high-dimensional goal observations into discrete, multi-factor representations. This approach significantly improves an agent's ability to generalize to unseen goals and provides a robust abstraction for hierarchical reinforcement learning.

What Problem Does This Paper Address?

Goal-conditioned RL faces two major hurdles:

  1. Goal Grounding: How does an agent relate a raw sensory input (like a 64x64 pixel image) to a specific objective? Continuous embeddings often capture too much noise, making it hard for the agent to distinguish between "important" features and background clutter.
  2. Goal Specification & Generalization: How do we tell an agent to reach a goal it has never seen before? If goals are represented as continuous vectors, "out-of-distribution" goals often lead to unpredictable agent behavior.

The authors argue that the world is naturally compositional (e.g., an object has a shape and a color). By forcing goal representations to be both discrete (categorical) and factorial (composed of multiple independent parts), agents can better understand the underlying structure of their tasks.

Methods

The core of the paper is the DGRL (Discrete Goal-conditioned RL) framework, which consists of three main stages:

The DGRL framework architecture
  • Self-Supervised Learning: The agent first learns to embed raw observations into a continuous space using standard techniques like Autoencoders or Deep InfoMax.
  • The Discretization Bottleneck: This is the "secret sauce." The continuous embedding is split into $G$ segments. Each segment is passed through a Vector Quantization (VQ) layer, where it is mapped to the closest entry in a learned "codebook" of discrete vectors.
  • Factorial Representation: By having multiple segments (factors), the agent represents a goal as a combination of discrete codes (e.g., Factor 1 = "Red", Factor 2 = "Square").
  • Downstream RL: These discrete codes are then fed into the RL policy. In hierarchical setups, a high-level policy picks these discrete codes as "sub-goals" for a low-level policy to execute.

The authors also provide a theoretical proof showing that this discretization creates a "concentration" effect, effectively lower-bounding the expected return when the agent encounters goals it didn't see during training.

Datasets

The researchers evaluated DGRL across a diverse set of benchmarks:

  • Color-MNIST: A toy dataset used to prove that the model successfully separates "shape" and "color" into different factors.
  • Gridworlds (Loopworld & SpiralWorld): Testing navigation and generalization to novel positions using pixel inputs.
  • MiniGrid: Procedurally generated environments used to test hard exploration.
  • KeyChest: A discrete maze requiring the agent to pick up keys to open chests.
  • Ant Manipulation & Navigation: Complex continuous control tasks (AntMaze, AntFall, AntPush) where an "Ant" robot must move objects or navigate mazes.
  • Sawyer Robotic Arm: A high-dimensional vision-based task where a robotic arm must move a puck to various target locations.

Key Insights and Ideas

  • Compositionality Emerges Naturally: In the Color-MNIST tests, the model learned to use one factor for the digit's shape and another for its color without being explicitly told to do so. This "factorial" nature is what allows for zero-shot generalization.
  • Discretization as a Denoising Filter: By forcing the representation into a fixed set of codes, the agent ignores small, irrelevant variations in the environment, focusing only on the "coarse" details necessary to reach the goal.
  • Better Intrinsic Rewards: The authors found that they could create a powerful intrinsic reward by simply measuring the fraction of discrete factors that match between the agent's current state and the goal. This proved more effective than standard distance metrics in continuous space.
  • More Factors = Better Performance: Generally, increasing the number of factors ($G$) improved results. Using 16 factors was often the "sweet spot" for complex tasks, providing enough expressivity while maintaining the benefits of discretization.

Key Results

  • Superior Generalization: In the SpiralWorld environment, DGRL agents were significantly more successful at reaching "unseen" goals compared to baselines using continuous representations.
  • State-of-the-Art Exploration: On the MiniGrid KCmedium task, DGRL combined with the AMIGO framework achieved a 96% success rate, outperforming popular exploration methods like RIDE (90%) and RND (89%).
  • Robotic Efficiency: In the Sawyer robotic arm task, DGRL reached high success rates much faster than the previous state-of-the-art (RIS), which struggled with the temporal reasoning required for the task.
  • Hierarchical Success: In "Ant" tasks, integrating DGRL into hierarchical frameworks (like HRAC) led to higher success rates in sparse-reward environments where the agent must perform a sequence of complex maneuvers.

Limitations and Future Work

While DGRL is powerful, the authors note a few areas for growth:

  • Semantic Meaning: While the factors are discrete, we don't always know exactly what each code represents semantically without manual inspection.
  • Codebook Collapse: Like many VQ-based methods, there is a risk that the model only uses a few entries in the codebook (though they used moving averages to mitigate this).
  • Future Direction: The authors suggest exploring how to explicitly enforce compositionality (the idea that the meaning of a complex expression is determined by its parts) even more strictly within the latent space.

For researchers in RL, this paper makes a strong case: if you want your agent to generalize and understand its goals better, sometimes it's better to be discrete.

Summary updated

Abstract

title> <link rel="stylesheet" href="/static/papers/css/papers.css" /> <meta name="citation_title" content="Discrete Compositional Representations as an Abstraction for Goal Conditioned Reinforcement Learning" /> <meta name="citation_author" content="Islam, Riashat" /> <meta name="citation_author" content="Zang, Hongyu" /> <meta name="citation_author" content="Goyal, Anirudh" /> <meta name="citation_author" content="Lamb, Alex M." /> <meta name="citation_author" content="Kawaguchi, Kenji" /> <meta name="citation_author" content="Li, Xin" /> <meta name="citation_author" content="Laroche, Romain" /> <meta name="citation_author" content="Bengio, Yoshua" /> <meta name="citation_author" content="Tachet des Combes, Remi" /> <meta name="citation_journal_title" content="Advances in Neural Information Processing Systems" /> <meta name="citation_volume" content="35" /> <meta name="citation_firstpage" content="3885" /> <meta name="citation_lastpage" content="3899" /> <meta name="citation_pdf_url" content="https://proceedings.neurips.cc/paper_files/paper/2022/file/18ddfb199d71a8a24f83abc1ced077b7-Paper-Conference.pdf" /> <meta name="citation_publication_date" content="2022-12-06" /><!-- Bootstrap CSS --> <!-- https://codepen.io/surjithctly/pen/PJqKzQ --> <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css" integrity="sha384-Vkoo8x4CGsO3+Hhxv8T/Q5PaXtkKtu6ug5TOeNV6gBiFeWPGFN9MuhOf23Q9Ifjh" crossorigin="anonymous" /> <link href="/static/menus/css/menus.css" rel="stylesheet" id="bootstrap-css" /> <link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.8.1/css/all.css" integrity="sha384-50oBUHEmvpQ+1lW4y57PTFmhCaXp0ML5d60M1M7uH2+nqUivzIebhndOJK28anvf" crossorigin="anonymous" /> <script type="text/javascript" async="async" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/latest.js?config=TeX-MML-AM_CHTML"></script> <script type="text/x-mathjax-config"> <![CDATA[ MathJax.Hub.Config({ "tex2jax": { "inlineMath": [["$","$"], ["\\(","\\)"]], "displayMath": [["\\[","\\]"]], "processEscapes": true } } ); ]]> </script> <style> <![CDATA[ @media (prefers-color-scheme: dark) { body { background-color: #333; color: #eee; } } .btn-spacer { margin: 2px; } .footer { position: fixed; left: 0; bottom: 0; width: 100%; background-color: #eee; color: black; } ]]> </style> <nav class="navbar navbar-expand-md navbar-light bg-light"> <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarToggler6" aria-controls="navbarToggler6" aria-expanded="false" aria-label="Toggle navigation"><span class="navbar-toggler-icon"></span></button> <div class="collapse navbar-collapse" id="navbarToggler6"> <a class="navbar-brand" href="/">NeurIPS Proceedings</a> <ul class="navbar-nav mr-auto mt-2 mt-md-0"> <li class="nav-item"> <a class="nav-link" href="/admin/login/?next=/admin/"><i class="fas fa-sign-in-alt" title="Login"></i></a> <li class="nav-item"> <a class="nav-link" href="/admin/logout/?nextp=/admin"><i class="fas fa-sign-out-alt" title="Logout"></i></a> <form class="form-inline my-2 my-lg-0" method="get" role="search" action="/papers/search"> <input class="form-control mr-sm-2" type="text" name="q" placeholder="Search" aria-label="Search" id="navsearch" /> <button class="btn btn-outline-success my-2 my-sm-0" type="submit">Search</button>
BibTeX
@inproceedings{islam2022discrete,
  author = {Riashat Islam and Hongyu Zang and Anirudh Goyal and Alex M. Lamb and Kenji Kawaguchi and Xin Li and Romain Laroche and Yoshua Bengio and Remi Tachet des Combes},
  title = {{Discrete Compositional Representations as an Abstraction for Goal Conditioned Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=sOSzuqvJhm},
}
Figure