Model-based Reinforcement Learning (RL) has achieved massive success with algorithms like MuZero, which can master complex games like Chess and Go without knowing the rules beforehand. However, MuZero relies on a critical assumption: the environment must be deterministic.
In the real world—and in complex simulators—environments are often stochastic (random) or partially observable (you cannot see everything at once). If an agent tries to plan using a deterministic model in a random world, it fails because it cannot account for the multiple possible outcomes of a single action.
To bridge this gap, researchers have introduced Vector Quantized Models (VQM), a framework that allows RL agents to plan in stochastic and partially-observable environments. By using discrete autoencoders (VQ-VAEs) to represent environment transitions as latent variables, VQM extends the success of MuZero-style planning to complex 3D worlds and unpredictable scenarios, allowing agents to "imagine" different possible futures and plan accordingly.
The VQM Architecture
The core innovation of the VQM framework consists of two main stages:
- State VQ-VAE: The model uses a Vector Quantized Variational Autoencoder to compress sequences of observations and actions into a set of discrete latent variables. These variables represent the "hidden" information about how the environment changes (e.g., what an opponent might do or what lies around a corner).
- Transition Model: A model is trained to predict these discrete latent codes, alongside rewards and values. Instead of predicting raw pixels, it predicts the next "code" in the sequence.
Planning with VQM-MCTS
To make decisions, the agent uses a modified version of Monte Carlo Tree Search (MCTS). Standard MCTS branches only on the agent's actions. VQM-MCTS introduces stochastic nodes: the search tree branches both on the agent's actions and on the discrete latent variables representing the environment's possible responses.
The authors propose several architectural variants for this planning process:
- VQHybrid: Interleaves agent actions and environment latent variables.
- VQPure: Plans entirely within the discrete latent space, decoupling planning from the raw action space.
- VQJumpy: Allows for "temporal abstraction," where the model plans several steps ahead at once.
Key Insights and Experimental Results
The researchers evaluated their approach on two distinct benchmarks: Chess (treating the opponent as a stochastic environment variable) and DeepMind Lab (a 3D environment characterized by large visual observations and partial observability).
The Environment as an "Actor"
By representing environment transitions as discrete codes, the authors essentially treat the environment as a second player in a game. This allows them to use minimax or expectimax search logic even in single-player tasks. In a single-player chess experiment where the opponent's moves were hidden, standard MuZero’s performance collapsed. VQM recovered this performance by learning to represent the opponent's likely moves as latent variables, effectively "re-discovering" the two-player nature of the game.
Discreteness is a Feature, Not a Bug
While many models use continuous latent variables, this paper argues that discrete codes are superior for planning. They prevent the "blurriness" often seen in long-term predictions from continuous VAEs and fit naturally into the branching structure of tree search.
Performance Gains
- Chess: In the single-player setting, VQM-MCTS significantly outperformed an offline version of MuZero. When using "worst-case" planning (assuming the environment is adversarial), VQM matched the performance of a full two-player MuZero that had access to all opponent data.
- DeepMind Lab: VQM produced high-quality, long-range rollouts (up to 100+ steps) without the typical "drifting" or blurring seen in standard LSTM-based predictors. It achieved a lower Mean Best Reconstruction Error (MBRE) than continuous VAE baselines, proving it better captures the multi-modal nature of complex 3D environments.
Limitations and Future Work
While highly effective, the current VQM framework relies on a two-stage training process where the VQ-VAE and the transition model are trained separately. Training the entire system end-to-end remains a promising but challenging future direction. Additionally, branching on both actions and latent variables increases the search space of MCTS, which could become computationally expensive in environments with very high entropy. Finally, while VQHybrid was the primary focus of the experiments, the authors note that "VQPure" and "VQJumpy" remain largely unexplored and could hold the key to even more efficient long-term planning.
Summary updated