Abstract
Deep model-based Reinforcement Learning (RL) has the potential to substantially improve the sample-efficiency of deep RL. While various challenges have long held it back, a number of papers have recently come out reporting success with deep model-based methods. This is a great development, but the lack of a consistent metric to evaluate such methods makes it difficult to compare various approaches. For example, the common single-task sample-efficiency metric conflates improvements due to model-based learning with various other aspects, such as representation learning, making it difficult to assess true progress on model-based RL. To address this, we introduce an experimental setup to evaluate model-based behavior of RL methods, inspired by work from neuroscience on detecting model-based behavior in humans and animals. Our metric based on this setup, the Local Change Adaptation (LoCA) regret, measures how quickly an RL method adapts to a local change in the environment. Our metric can identify model-based behavior, even if the method uses a poor representation and provides insight in how close a method's behavior is from optimal model-based behavior. We use our setup to evaluate the model-based behavior of MuZero on a variation of the classic Mountain Car task.
Introduction
In Reinforcement Learning (RL), a fundamental distinction exists between model-free methods, which learn reactive policies through trial and error, and model-based methods, which learn a predictive model of environment dynamics to plan ahead. While deep model-based RL has seen recent success (e.g., MuZero), evaluating these methods using standard single-task sample efficiency is fundamentally flawed.
Single-task efficiency conflates planning capabilities with representation learning and exploration strategies. A model-free agent with a superior representation might outperform a model-based agent with a poor one, masking the true nature of the algorithm. To address this, we introduce the Local Change Adaptation (LoCA) regret, a metric that explicitly tests for the defining characteristic of a model-based system: the ability to instantly update a global plan when a local piece of the world changes.
The LoCA Setup
Inspired by neuroscience experiments used to detect deliberative planning in animals, the LoCA setup involves three distinct phases designed to isolate planning from habit-based learning:
- Phase 1 (Global Pretraining): The agent is trained on Task A until it converges on an optimal policy (e.g., reaching Terminal 1 for a high reward).
- Phase 2 (Local Pretraining): The agent is moved to Task B, where the reward at Terminal 1 has decreased. Crucially, the agent is restricted to the area immediately surrounding Terminal 1 using a "one-way passage," preventing it from exploring the rest of the environment.
- Phase 3 (Evaluation): The agent is placed in the middle of the environment in Task B.
An ideal model-based agent updates its internal model during Phase 2 and immediately plans a new route to an alternative goal (Terminal 2) in Phase 3, resulting in a LoCA regret of 0. Conversely, a model-free agent relies on cached values and must unlearn its habit through trial and error, resulting in high regret.
Key Results and Insights
Tabular and Deep RL Performance
In tabular environments, an ideal Model-Based Value Iteration (MB-VI) agent achieves exactly 0.00 LoCA regret, adapting instantly. In contrast, model-free Sarsa($\lambda$) requires significant time to unlearn its previous policy. This disparity is clearly visible when tracking the fraction of episodes where the agent reaches the new optimal terminal.
When applied to deep RL in a modified Mountain Car task, we found that MuZero, while outperforming purely model-free approaches, does not exhibit ideal model-based behavior. MuZero achieved a LoCA regret of 2.24, compared to Sarsa($\lambda$)'s 7.30, suggesting it behaves as a hybrid system rather than a pure planner.
The On-Policy Trap
Our analysis revealed that adding on-policy elements, such as $n$-step bootstrapping, can be detrimental to model-based adaptability. While these elements often improve single-task benchmarks, they cripple the agent's ability to adapt to local changes. In tabular tests, a 1-step model had a LoCA regret of 1.68, which ballooned to 34.77 when using a 5-step model. This suggests that current optimization for single-task efficiency may be steering research away from true model-based flexibility.
Conclusion
The LoCA regret provides a consistent, representation-agnostic metric for evaluating true model-based behavior. Our results indicate that even state-of-the-art methods like MuZero have significant room for improvement to reach the adaptability of ideal model-based systems. Future work will extend this metric to more complex, stochastic environments to further refine our understanding of deep model-based RL.
Summary updated