Recurrent World Models Facilitate Policy Evolution

David Ha, Jürgen Schmidhuber

Introduction

Traditional Reinforcement Learning (RL) agents often struggle with two major challenges: processing high-dimensional sensory input (like raw pixels) and handling long-term credit assignment. Processing every pixel in every frame is computationally expensive, and determining which specific action led to a reward minutes later is mathematically complex. Furthermore, most RL agents require millions of interactions with the actual environment to learn, which can be slow and potentially dangerous in real-world applications.

To address this, the authors introduce "World Models," a framework where an agent learns a compressed, generative representation of its environment (a "dream") and then trains a simple policy entirely within that simulated reality. By decoupling world-learning from decision-making, the authors achieved state-of-the-art results in RL tasks, including the first-ever "solution" to the challenging CarRacing-v0 benchmark.

The World Model Architecture

The authors propose a modular architecture inspired by the human cognitive system, consisting of three distinct components:

  1. Vision (V): A Variational Autoencoder (VAE) compresses high-dimensional 2D image frames into a small, representative latent vector ($z$). This handles the "spatial" aspect of the world, extracting essential features while discarding redundant pixel data.
  2. Memory (M): A Recurrent Neural Network (RNN) combined with a Mixture Density Network (MDN). This "MDN-RNN" predicts the probability distribution of the next latent vector ($z_{t+1}$) based on past observations and actions. This handles the "temporal" aspect and accounts for the inherent randomness (stochasticity) of the environment.
  3. Controller (C): A very simple, single-layer linear model that maps the current vision ($z$) and memory ($h$) to an action. Because the Vision and Memory models do the heavy lifting of understanding the world, the Controller can be remarkably small (less than 1,000 parameters) and is trained using Evolution Strategies (CMA-ES) rather than traditional backpropagation.

Dream Training

A key innovation of this paper is the "Dream Training" setup. The agent is trained inside a hallucinated version of the environment generated entirely by its own Memory (M) model.

Hallucinated "dream" environments generated by the agent's internal world model.

To prevent the agent from "cheating" by exploiting deterministic flaws in its own model, the authors introduced a temperature parameter ($\tau$) to add uncertainty and noise to the hallucinated environment. This forces the agent to learn a robust policy that can survive unexpected scenarios.

Environments and Evaluation

The model was evaluated on two distinct environments from the OpenAI Gym:

  • CarRacing-v0: A top-down racing task where the agent must navigate randomly generated tracks.
  • VizDoom (DoomTakeCover-v0): A 3D environment where the agent must dodge fireballs launched by monsters to survive as long as possible.
Actual environments for CarRacing-v0 and VizDoom.

Key Findings and Results

  • State-of-the-Art Performance: On the CarRacing-v0 task, the World Model approach achieved a score of 906 ± 21, effectively "solving" the environment (defined as a score > 900). At the time, this was the first reported solution to this task using raw pixel inputs.
  • Successful Transfer from Dreams to Reality: For VizDoom, the agent was trained entirely inside its own hallucinated dream. When transferred to the actual VizDoom game, it achieved a score of 1092 ± 556, far exceeding the "solution" requirement of 750 and outperforming agents trained directly on the actual game.
  • The Power of Decoupling: By training the Vision and Memory components in an unsupervised way (just predicting the next frame), the agent learns "what the world looks like" without needing a reward signal. This leaves the Controller with a much simpler task.
  • Predictive Coding: The RNN's hidden state ($h$) acts as a "prediction" of the future. Giving the controller access to these internal "expectations" significantly improves performance compared to just looking at the current frame.
  • Computational Efficiency: Because the Controller is so small, it could be evolved on a single machine with multiple CPU cores in a matter of days, rather than requiring massive GPU clusters.

Limitations and Future Work

While highly effective, the framework has limitations. The VAE is unsupervised and tries to reconstruct everything in the frame, meaning it might waste capacity modeling task-irrelevant information (like background clouds) that have no impact on gameplay. Additionally, while the LSTM-based world model is powerful, it has a finite memory. For much larger, open-world environments, the model might suffer from "catastrophic forgetting" or lack the capacity to store the entire environment's logic.

In this paper, the world model was trained once on random data. For more complex tasks, the authors suggest an iterative approach—where the agent explores, updates its world model, and then dreams again—would be necessary to master increasingly difficult domains.

Summary updated

Abstract

A generative recurrent neural network is quickly trained in an unsupervised manner to model popular reinforcement learning environments through compressed spatio-temporal representations. The world model's extracted features are fed into compact and simple policies trained by evolution, achieving state of the art results in various environments. We also train our agent entirely inside of an environment generated by its own internal world model, and transfer this policy back into the actual environment. Interactive version of this paper is available at https://worldmodels.github.io
BibTeX
@inproceedings{ha2018recurrent,
  author = {David Ha and Jürgen Schmidhuber},
  title = {{Recurrent World Models Facilitate Policy Evolution}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=B1EKjDbubr},
}
Figure