In reinforcement learning (RL), agents generally fall into two camps: model-free (learning directly from trial and error) and model-based (learning a simulator of the environment to "plan" in). Model-based RL is often touted as more data-efficient because the agent can practice in its "head" without needing more real-world interactions.
However, this paper from DeepMind researchers explores a fundamental question: When should we bother learning a parametric "world model" versus simply replaying past experiences? By re-evaluating the strengths of experience replay and proposing smarter ways to use learned models, the authors demonstrate that a well-tuned model-free agent can actually outperform state-of-the-art model-based systems in data-efficient settings.
The Problem with Perfect Worlds
Learning a perfect model of a complex world, like a video game, is incredibly difficult. If the model is even slightly wrong, planning with it can lead the agent astray. This instability is often linked to the "Deadly Triad"—the divergence caused by combining function approximation, bootstrapping, and off-policy learning.
This paper investigates whether the perceived benefits of model-based RL are actually due to the models themselves, or simply the extra computation (planning) they allow. Crucially, experience replay can also provide this extra computation without the risk of model hallucinations.
Replay as a Non-Parametric Model
The authors provide a unified framework treating both experience replay and parametric models as forms of planning. A key insight is that experience replay is essentially a perfect, non-parametric model of the states the agent has already seen. It doesn't suffer from the drift that a neural network world model might.
The authors prove that uniform experience replay is inherently stable under certain conditions, whereas a learned model can easily cause the agent's value estimates to diverge because the model's errors create a distribution shift.
To test this in practice, the researchers modified a standard Rainbow DQN (a model-free agent) by increasing multi-step returns and starting replay much earlier. This "Data-Efficient Rainbow" was able to compete in the low-data regime (100k interactions), achieving 25% higher median performance than the leading model-based agent (SimPLe) on the Atari benchmark, outperforming it in 17 out of 26 games.
Smarter Ways to Plan: Backward and "In the Now"
If replay is so strong, when should we use learned models? The paper introduces two highly effective strategies tested in controlled Grid World environments:
- Backward Planning for Credit Assignment: Instead of using a model to predict the next state (forward), the authors used it to predict the previous state (backward). The logic is sound: if a model is wrong, updating a "fictional" previous state is less harmful than corrupting the value of a "real" current state. If your model predicts a magical path to a reward that doesn't exist, a forward-planning agent will incorrectly value its current real state. A backward-planning agent only incorrectly values a fictional, unreachable state.
- Planning for Behavior ("In the Now"): Forward planning is most useful when used to search for the best immediate move (like a chess engine evaluating the next few turns), rather than trusting those imagined steps to update the agent's permanent policy or long-term knowledge.
In grid world experiments, backward planning led to faster and more stable learning than forward planning, especially when the environment was stochastic. Furthermore, planning "in the now" significantly reduced the total steps required to reach the goal.
Limitations and Future Work
While these findings strongly advocate for the power of experience replay and targeted planning, challenges remain. Much of the theoretical stability analysis in the paper is grounded in linear function approximation, and extending these proofs to deep neural networks is an ongoing challenge.
Additionally, the paper focuses on "pixel-in, action-out" models. The authors suggest that "abstract models"—which plan in a latent feature space rather than raw pixels—might bridge the gap between model-free and model-based efficiency. Finally, for non-parametric models like replay to generalize better, future research must develop better ways to define the "distance" between high-dimensional states.
Summary updated