Vector Quantized Models for Planning

Sherjil Ozair, Yazhe Li, Ali Razavi, Ioannis Antonoglou, Aäron van den Oord, Oriol Vinyals

Abstract

Recent developments in the field of model-based RL have proven successful in a range of environments, especially ones where planning is essential. However, such successes have been limited to deterministic fully-observed environments. We present a new approach that handles stochastic and partially-observable environments. Our key insight is to use discrete autoencoders to capture the multiple possible effects of an action in a stochastic environment. We use a stochastic variant of Monte Carlo tree search to plan over both the agent's actions and the discrete latent variables representing the environment's response. Our approach significantly outperforms an offline version of MuZero on a stochastic interpretation of chess where the opponent is considered part of the environment. We also show that our approach scales to DeepMind Lab, a first-person 3D environment with large visual observations and partial observability.

Introduction

Model-based Reinforcement Learning (RL) has achieved remarkable milestones with algorithms like MuZero, which can master complex games like Chess and Go without prior knowledge of the rules. However, a significant limitation of these models is their reliance on the assumption of a deterministic environment. In real-world scenarios and complex simulations, environments are often stochastic and partially observable. Without the ability to predict various potential environment reactions, an agent cannot construct an effective look-ahead tree. While continuous latent variables have been explored to address this, they are difficult to integrate with discrete search algorithms like Monte Carlo Tree Search (MCTS).

Methods: Vector Quantized Models (VQM)

The proposed Vector Quantized Models (VQM) framework treats the environment's response as a discrete latent variable, allowing it to be searched alongside the agent's actions.

Core Components

  1. State VQ-VAE: The model utilizes Vector Quantized Variational Autoencoders (VQ-VAEs) to compress sequences of observations and actions into a discrete latent space. This quantization transforms high-dimensional environment transitions into a finite set of "codes" from a learned codebook.
  2. Transition Model: A predictive model is trained to forecast the next discrete latent code, reward, and value based on the current state and action.
  3. MCTS over Latents: During the planning phase, MCTS branches not only on the agent's possible actions ($a$) but also on the environment's possible latent responses ($k$). This enables the agent to simulate various ways the world might react to its decisions.

Architectural Variants

The research explores several ways to structure this search:

  • VQHybrid: Interleaves agent actions and environment latent codes.
  • VQPure: Decouples search from the raw action space by planning entirely within the latent space.
  • VQJumpy: Facilitates temporal abstraction by predicting states multiple steps into the future.
Comparison of planning architectures across different models.

Experimental Setup

The VQM framework was evaluated on two distinct benchmarks:

  • Stochastic Chess: Using the Million Base and FICS datasets, the authors created a "single-player" version of chess. Here, the opponent is treated as a stochastic element of the environment dynamics rather than a known second player.
  • DeepMind Lab: To test scalability to complex visual observations and partial observability, the model was trained on over 100 million episodes from a 3D first-person navigation task.

Key Results

Performance in Stochastic Environments

In the single-player chess setup, standard MuZero performance collapses because it cannot account for the unobserved opponent's moves. VQM (specifically VQHybrid and VQPure) successfully recovers this performance, matching the results of a two-player MuZero that has full access to opponent actions. This demonstrates that VQM effectively captures the "environment actions" (opponent moves) within its latent codes.

Visual Prediction and Scaling

In DeepMind Lab, VQM outperformed LSTM and continuous VAE baselines in long-term rollouts. Using Mean Best Reconstruction Error (MBRE) as a metric, VQM demonstrated a superior ability to capture the multi-modal nature of 3D environments—for instance, correctly imagining different possible turns in a hallway. This confirms that VQ-based planning is effective for high-dimensional visual inputs, not just symbolic games.

Conclusion and Future Work

VQM provides a robust framework for planning in stochastic and partially observable worlds by discretizing environment transitions. While the current implementation uses a two-stage training process, future work aims to explore end-to-end training. Additionally, while branching on latent codes increases search complexity, the promise of temporal abstraction through "Jumpy" and "Pure" variants offers a compelling path for future research in hierarchical planning.

Summary updated

Abstract

Recent developments in the field of model-based RL have proven successful in a range of environments, especially ones where planning is essential. However, such successes have been limited to deterministic fully-observed environments. We present a new approach that handles stochastic and partially-observable environments. Our key insight is to use discrete autoencoders to capture the multiple possible effects of an action in a stochastic environment. We use a stochastic variant of Monte Carlo tree search to plan over both the agent's actions and the discrete latent variables representing the environment's response. Our approach significantly outperforms an offline version of MuZero on a stochastic interpretation of chess where the opponent is considered part of the environment. We also show that our approach scales to DeepMind Lab, a first-person 3D environment with large visual observations and partial observability.
BibTeX
@inproceedings{ozair2021vector,
  author = {Sherjil Ozair and Yazhe Li and Ali Razavi and Ioannis Antonoglou and Aäron van den Oord and Oriol Vinyals},
  title = {{Vector Quantized Models for Planning}},
  booktitle = {International Conference on Machine Learning},
  year = {2021},
  url = {https://openreview.net/forum?id=aQNudGfjay},
}
Figure