When to use parametric models in reinforcement learning?

Hado van Hasselt, Matteo Hessel, John Aslanides

In reinforcement learning (RL), agents generally fall into two camps: model-free (learning directly from trial and error) and model-based (learning a simulator of the environment to "plan" in). Model-based RL is often touted as more data-efficient because the agent can practice in its "head" without needing more real-world interactions.

However, this paper from DeepMind researchers explores a fundamental question: When should we bother learning a parametric "world model" versus simply replaying past experiences? By re-evaluating the strengths of experience replay and proposing smarter ways to use learned models, the authors demonstrate that a well-tuned model-free agent can actually outperform state-of-the-art model-based systems in data-efficient settings.

The Problem with Perfect Worlds

Learning a perfect model of a complex world, like a video game, is incredibly difficult. If the model is even slightly wrong, planning with it can lead the agent astray. This instability is often linked to the "Deadly Triad"—the divergence caused by combining function approximation, bootstrapping, and off-policy learning.

This paper investigates whether the perceived benefits of model-based RL are actually due to the models themselves, or simply the extra computation (planning) they allow. Crucially, experience replay can also provide this extra computation without the risk of model hallucinations.

Replay as a Non-Parametric Model

The authors provide a unified framework treating both experience replay and parametric models as forms of planning. A key insight is that experience replay is essentially a perfect, non-parametric model of the states the agent has already seen. It doesn't suffer from the drift that a neural network world model might.

The authors prove that uniform experience replay is inherently stable under certain conditions, whereas a learned model can easily cause the agent's value estimates to diverge because the model's errors create a distribution shift.

Scalability of Replay vs. Forward Dyna

To test this in practice, the researchers modified a standard Rainbow DQN (a model-free agent) by increasing multi-step returns and starting replay much earlier. This "Data-Efficient Rainbow" was able to compete in the low-data regime (100k interactions), achieving 25% higher median performance than the leading model-based agent (SimPLe) on the Atari benchmark, outperforming it in 17 out of 26 games.

Smarter Ways to Plan: Backward and "In the Now"

If replay is so strong, when should we use learned models? The paper introduces two highly effective strategies tested in controlled Grid World environments:

  1. Backward Planning for Credit Assignment: Instead of using a model to predict the next state (forward), the authors used it to predict the previous state (backward). The logic is sound: if a model is wrong, updating a "fictional" previous state is less harmful than corrupting the value of a "real" current state. If your model predicts a magical path to a reward that doesn't exist, a forward-planning agent will incorrectly value its current real state. A backward-planning agent only incorrectly values a fictional, unreachable state.
  2. Planning for Behavior ("In the Now"): Forward planning is most useful when used to search for the best immediate move (like a chess engine evaluating the next few turns), rather than trusting those imagined steps to update the agent's permanent policy or long-term knowledge.
Grid world experiments demonstrating the efficiency of backward planning and planning in the now

In grid world experiments, backward planning led to faster and more stable learning than forward planning, especially when the environment was stochastic. Furthermore, planning "in the now" significantly reduced the total steps required to reach the goal.

Limitations and Future Work

While these findings strongly advocate for the power of experience replay and targeted planning, challenges remain. Much of the theoretical stability analysis in the paper is grounded in linear function approximation, and extending these proofs to deep neural networks is an ongoing challenge.

Additionally, the paper focuses on "pixel-in, action-out" models. The authors suggest that "abstract models"—which plan in a latent feature space rather than raw pixels—might bridge the gap between model-free and model-based efficiency. Finally, for non-parametric models like replay to generalize better, future research must develop better ways to define the "distance" between high-dimensional states.

Summary updated

Abstract

We examine the question of when and how parametric models are most useful in reinforcement learning. In particular, we look at commonalities and differences between parametric models and experience replay. Replay-based learning algorithms share important traits with model-based approaches, including the ability to plan: to use more computation without additional data to improve predictions and behaviour. We discuss when to expect benefits from either approach, and interpret prior work in this context. We hypothesise that, under suitable conditions, replay-based algorithms should be competitive to or better than model-based algorithms if the model is used only to generate fictional transitions from observed states for an update rule that is otherwise model-free. We validated this hypothesis on Atari 2600 video games. The replay-based algorithm attained state-of-the-art data efficiency, improving over prior results with parametric models. Additionally, we discuss different ways to use models. We show that it can be better to plan backward than to plan forward when using models to perform credit assignment (e.g., to directly learn a value or policy), even though the latter seems more common. Finally, we argue and demonstrate that it can be beneficial to plan forward for immediate behaviour, rather than for credit assignment.
BibTeX
@inproceedings{hasselt2019when,
  author = {Hado van Hasselt and Matteo Hessel and John Aslanides},
  title = {{When to use parametric models in reinforcement learning?}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=IELYc2ZArZR},
}
Figure