Abstract
We examine the question of when and how parametric models are most useful in reinforcement learning. In particular, we look at commonalities and differences between parametric models and experience replay. Replay-based learning algorithms share important traits with model-based approaches, including the ability to plan: to use more computation without additional data to improve predictions and behaviour. We discuss when to expect benefits from either approach, and interpret prior work in this context. We hypothesise that, under suitable conditions, replay-based algorithms should be competitive to or better than model-based algorithms if the model is used only to generate fictional transitions from observed states for an update rule that is otherwise model-free. We validated this hypothesis on Atari 2600 video games. The replay-based algorithm attained state-of-the-art data efficiency, improving over prior results with parametric models.
Introduction
Reinforcement learning (RL) agents are notoriously data-hungry, often requiring millions of interactions to learn simple tasks. To solve this, researchers typically look to Model-Based RL, where an agent learns a parametric model (a "simulator") of its environment to "plan" or "dream" up extra experiences. However, there is another way to gain efficiency: Experience Replay, where the agent stores and re-watches its past real-world interactions. This paper investigates a fundamental question: When is a learned parametric model actually better than just replaying old data? By identifying commonalities between these two approaches, we highlight scenarios where models might introduce errors that outweigh their benefits.
Methodology
We provide a unifying framework that describes both model-based and replay-based RL as variations of the same process. Key methodological contributions include:
- Theoretical Analysis of Stability: Using the "Deadly Triad" theory (function approximation, bootstrapping, and off-policy learning), we demonstrate that parametric models are inherently more prone to divergence than experience replay.
- Data-Efficient Rainbow: We modified the standard Rainbow DQN (a model-free agent) by increasing multi-step returns (from 3 to 20 steps) and increasing the frequency of updates relative to environment steps.
- Planning Variations: We experimented with "Forward Dyna" (planning into the future), "Backward Dyna" (planning from the future back to the past), and "Planning in the Now" (look-ahead search to choose the immediate next action).
Theoretical Insights
Replay as a Perfect Model
Experience replay can be viewed as a non-parametric model that is perfectly accurate for states the agent has already visited. While a learned parametric model attempts to generalize to new states, this generalization often introduces errors.
The Stability Gap
We prove that replaying full episodes is guaranteed to be stable. In contrast, using a learned model to generate transitions from a distribution that doesn't match the model's own dynamics can cause the agent's value estimates to diverge.
Planning vs. Credit Assignment
Parametric models are most valuable for "planning in the now"—using a search tree to decide the immediate next action. They are less essential for "credit assignment" (updating the values of past states), where experience replay is often safer and more efficient.
Experimental Results
Grid World Analysis
In grid-world experiments, we found that planning backward from a known state was more robust than planning forward. If a model is inaccurate, updating an "imagined" predecessor state is less damaging than updating a "real" state with a fake future.
Atari Benchmark
The most striking result is the head-to-head comparison on the Atari benchmark:
- Superior Efficiency: The "Data-Efficient Rainbow" (model-free) matched the performance of SimPLe (model-based) after only 70,000 interactions.
- Higher Ceiling: By 100,000 interactions, Rainbow reached a median human-normalized score roughly 25% higher than SimPLe.
- Win Rate: Rainbow outperformed the model-based agent in 17 out of 26 games.
- Computational Savings: Sampling from a replay buffer is orders of magnitude faster than generating "imagined" frames from a deep neural network world model.
Limitations and Future Work
The study focuses on Atari, where states are high-dimensional pixels. In environments requiring generalization to never-before-seen states, parametric models might still hold an edge. Future work could explore "abstract models" that predict future features or rewards rather than raw pixels, potentially avoiding stability issues. Additionally, non-parametric interpolation (e.g., kernel-based methods) could allow experience replay to generalize without the instability of parametric models.
Summary updated