Recurrent World Models Facilitate Policy Evolution

David Ha, Jürgen Schmidhuber

TL;DR

This paper introduces a novel approach to reinforcement learning (RL) by training agents within a learned "world model" of their environment. This allows for efficient policy learning and even enables training entirely within a simulated environment, which can then be transferred to the real world, achieving state-of-the-art results on challenging tasks.

What Problem Does This Paper Address?

Traditional reinforcement learning often requires agents to interact extensively with a real environment to learn effective policies. This can be computationally expensive, time-consuming, and sometimes dangerous. While model-based RL aims to mitigate this by learning a model of the environment, these models are often complex and difficult to train, or they are used in conjunction with real-world training, not as a complete replacement. The authors address the challenge of efficiently learning robust policies by proposing a method that uses a compressed, generative recurrent neural network (RNN) as a world model, and then trains a simple controller using this model, even to the extent of training entirely within the generated environment.

Methods

The core of the proposed method is a modular agent architecture consisting of three components:

  • Vision (V): A Variational Autoencoder (VAE) that compresses high-dimensional visual inputs (like images) into a low-dimensional latent vector z. This component learns to represent the current state efficiently.
  • Memory/World Model (M): A Mixture Density Network (MDN) combined with a Recurrent Neural Network (RNN), specifically an LSTM. This component learns to predict the next latent state z_{t+1} given the current latent state z_t, the action taken a_t, and the RNN's hidden state h_t. Crucially, it models a probability distribution over future states, making it robust to stochasticity. This MDN-RNN is trained unsupervisedly on data collected from the environment.
  • Controller (C): A simple, compact policy network (e.g., a linear model) that takes the latent state z_t and the RNN's hidden state h_t as input and outputs an action a_t. The simplicity of C makes it amenable to training via evolution strategies (like CMA-ES), which requires only the final reward signal.

A key innovation is the ability to train the controller C entirely within the generated environment produced by M. To prevent the controller from "cheating" or exploiting imperfections in the learned world model, the authors introduce a "temperature" parameter τ in the MDN-RNN's sampling process. Increasing τ adds more uncertainty to the generated environment, forcing the controller to learn more robust policies that generalize better to the actual environment.

Datasets

The paper evaluates its approach on two OpenAI Gym environments:

  • CarRacing-v0: A continuous control task where the agent learns to drive a car on randomly generated tracks.
  • DoomTakeCover-v0: A partially observable environment where the agent must survive by dodging fireballs.

The training data for the world model M was collected using random policies within these environments. The authors also introduce a "DoomRNN" environment, which is a computationally efficient, latent-space simulation of DoomTakeCover-v0, generated by their trained world model.

Key Insights and Ideas

  • World Models as Efficient Simulators: A compressed, generative world model can effectively replace the need for extensive real-world interaction, significantly speeding up RL training.
  • Unsupervised Learning for World Models: Training the world model M unsupervisedly on raw observations is sufficient for it to capture the environment's dynamics.
  • Simple Controllers, Complex World: By making the controller C simple and the world model M complex, the credit assignment problem for C is simplified, allowing for more effective training, especially with evolutionary methods.
  • Temperature for Robustness: Introducing uncertainty (via the temperature parameter τ) into the generated environment during training is crucial for preventing the agent from exploiting model imperfections and for improving generalization to the real environment.
  • Sim-to-Real Transfer: Policies trained entirely within a learned, generative world model can be successfully transferred to the actual environment, achieving state-of-the-art performance.

Key Results

  • CarRacing-v0: The proposed "Full World Model" approach achieved an average score of 906 ± 21, surpassing previous state-of-the-art results on this task. This significantly outperformed agents that only used the VAE's latent representation (z-only) without the RNN's temporal memory.
  • DoomTakeCover-v0 (via DoomRNN): An agent trained entirely within the generated DoomRNN environment achieved an average survival time of 1092 ± 556 when deployed to the actual VizDoom environment. This score was significantly higher than the required threshold and also surpassed scores achieved by agents trained in more difficult virtual environments (higher τ).

Limitations and Future Work

  • VAE Limitations: The VAE, trained unsupervisedly, might encode irrelevant information or fail to capture task-critical details. The authors note that training V and M jointly, potentially with reward prediction, could focus V on task-relevant features.
  • World Model Capacity: The LSTM-based world model has limitations in storing long-term historical information, potentially suffering from catastrophic forgetting. Future work could explore hypernetworks or external memory modules.
  • Iterative Training: For more complex environments, an iterative training procedure where the agent actively explores to improve its world model is suggested. Incorporating intrinsic motivation and curiosity is also proposed.
  • Generalization of World Models: The authors acknowledge that their current world model is task-specific. Future work could explore more general approaches, like those that learn to "think" or incorporate hierarchical planning.

Summary updated

Abstract

A generative recurrent neural network is quickly trained in an unsupervised manner to model popular reinforcement learning environments through compressed spatio-temporal representations. The world model's extracted features are fed into compact and simple policies trained by evolution, achieving state of the art results in various environments. We also train our agent entirely inside of an environment generated by its own internal world model, and transfer this policy back into the actual environment. Interactive version of paper at https://worldmodels.github.io
BibTeX
@inproceedings{ha2018recurrent,
  author = {David Ha and Jürgen Schmidhuber},
  title = {{Recurrent World Models Facilitate Policy Evolution}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=B1EKjDbubr},
}
Figure