Learning to Predict Without Looking Ahead: World Models Without Forward Prediction

C. Daniel Freeman, David Ha, Luke Metz

Introduction

In traditional model-based Reinforcement Learning (RL), agents typically learn a "world model" through a supervised loss: the model observes the current state and action, and is penalized if its prediction of the next state deviates from reality. While effective, this approach is computationally expensive and often forces the model to waste capacity on irrelevant environmental details.

However, in nature, biological brains likely evolved predictive capabilities as a byproduct of survival, rather than by optimizing a specific mathematical prediction loss. This paper investigates whether a functional world model can emerge simply as a side effect of solving an RL task under severe sensory constraints, without ever explicitly training the agent to predict the future.

Observational Dropout

To test this hypothesis, the authors introduce a training regime called observational dropout.

At every timestep, the agent is only allowed to observe the "ground truth" environment with a specific probability $p$ (the "peek probability"). When the agent is denied a view of the real world (probability $1-p$), it must rely on the output of its own internal model from the previous step as its current input.

Instead of training the predictive model and the action policy separately, both are trained simultaneously using a population-based version of REINFORCE. Crucially, there is no "prediction error" term in the objective function. The only signal the system receives is the standard task reward. To succeed while effectively "blind" most of the time, the model is forced to learn to generate a sequence of internal observations—hallucinations—that are accurate enough for the policy to make correct decisions.

Experimental Environments

The researchers evaluated this approach across three distinct environments:

  • Swing-up Cart-pole: A non-linear control task requiring an agent to swing a hanging pole upright and balance it.
  • Grid World: A 2D environment where a stateless agent navigates to collect rewards and avoid penalties.
  • Car Racing: A high-dimensional pixel-based environment where a pre-trained Variational Autoencoder (VAE) compresses images into a latent vector for the world model to process.

Key Insights: Function Over Form

The models that emerge from observational dropout are not perfect simulators of reality. Because the policy and model are trained jointly, the model only learns the dynamics strictly necessary for the task, and the policy learns to accommodate the model's specific "hallucinations."

Policy behavior in hallucinated vs. real environments

For instance, the model might learn to balance a pole at a slight angle rather than perfectly vertical. This demonstrates a "just enough" principle: the model ignores irrelevant physics (like the forward motion of a car on a straightaway) and focuses on critical information (like upcoming curves).

Furthermore, architectural inductive biases play a massive role in this emergent learning. In Grid World experiments, a simple Convolutional Neural Network (CNN) vastly outperformed a high-capacity Fully Connected model, as the CNN's inherent spatial translation allowed it to easily grasp the logic of movement.

Key Results and Robustness

The most striking result is the agents' robustness to extreme data loss. In the Swing-up Cart-pole task, agents successfully solved the environment even when they only observed the real world 5% to 10% of the time.

Mean cumulative reward vs. peek probability in Cart-pole Swing-up

Similarly, in the Car Racing environment, the agent maintained successful performance while being blind for 70% of the frames. Remarkably, world models trained via observational dropout (at very low peek probabilities) were often better at training new policies from scratch than models of the same size trained with a standard, explicit forward-prediction loss.

Limitations and Future Work

While observational dropout successfully coerces the emergence of world models, it is not without limitations. Because the model is not explicitly grounded in reality at every step, errors can compound over long "imagined" trajectories, eventually causing the agent's internal state to diverge from the true environment. Additionally, the current reliance on evolutionary strategies like REINFORCE is computationally heavy compared to gradient-based supervised learning. Future work will need to explore how to scale this emergent modeling to highly stochastic, complex multi-object environments while improving optimization efficiency.

Summary updated

Abstract

Much of model-based reinforcement learning involves learning a model of an agent's world, and training an agent to leverage this model to perform a task more efficiently. While these models are demonstrably useful for agents, every naturally occurring model of the world of which we are aware---e.g., a brain---arose as the byproduct of competing evolutionary pressures for survival, not minimization of a supervised forward-predictive loss via gradient descent. That useful models can arise out of the messy and slow optimization process of evolution suggests that forward-predictive modeling can arise as a side-effect of optimization under the right circumstances. Crucially, this optimization process need not explicitly be a forward-predictive loss. In this work, we introduce a modification to traditional reinforcement learning which we call observational dropout, whereby we limit the agents ability to observe the real environment at each timestep. In doing so, we can coerce an agent into learning a world model to fill in the observation gaps during reinforcement learning. We show that the emerged world model, while not explicitly trained to predict the future, can help the agent learn key skills required to perform well in its environment. Videos of our results available at https://learningtopredict.github.io/
BibTeX
@inproceedings{freeman2019learning,
  author = {C. Daniel Freeman and David Ha and Luke Metz},
  title = {{Learning to Predict Without Looking Ahead: World Models Without Forward Prediction}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=yyFKJO0ySQg},
}
Figure