TL;DR
The LoCA (Local Change Adaptation) regret is a new metric designed to isolate and evaluate "model-based" behavior in Reinforcement Learning (RL) agents. By measuring how quickly an agent adapts to a local change in its environment, LoCA reveals whether an algorithm is truly planning with a world model or simply benefiting from better representations—showing that even state-of-the-art models like MuZero still fall short of ideal model-based behavior.
The Problem: Disentangling Planning from Habit
In Reinforcement Learning, we often distinguish between model-free methods (which learn "habits" through trial and error) and model-based methods (which learn a "map" or model of the world to plan ahead).
The current gold standard for evaluating model-based RL is "single-task sample efficiency"—essentially, how fast an agent reaches a high score. However, this is a flawed metric for isolating model-based behavior. A model-free agent might appear highly sample-efficient simply because it has a superior neural network architecture or a clever exploration strategy, not because it is actually "planning."
Without a way to isolate model-based behavior, it is difficult to measure true progress in the field. We need to know: Is the agent actually using its model to adapt to changes, or is it just a very efficient habit-learner?
The LoCA Methodology
To address this, the authors introduce the Local Change Adaptation (LoCA) setup, inspired by neuroscience experiments used to detect deliberative behavior in animals.
The Experimental Setup
The setup involves two tasks, Task A and Task B, which are identical except for a local change in rewards or dynamics.
- Phase 1 (Pretraining): The agent learns Task A, where a high reward is located at Terminal State 1 (T1).
- Phase 2 (Local Adaptation): The agent is placed in a restricted area of Task B where it discovers that the reward at T1 has been significantly reduced. Crucially, it cannot see the rest of the environment during this phase.
- Phase 3 (Evaluation): The agent is released into the full environment of Task B.
The Metric: LoCA Regret
An ideal model-based agent should update its internal model during Phase 2 and immediately "realize" (via planning) that it should now head toward a different goal (T2) when released in Phase 3. A model-free agent, however, would still try to go to T1 until it physically experiences the failure multiple times in the full environment.
LoCA Regret quantifies this delay: it is the sum of the performance gap between the agent and an optimal model-based agent during the adaptation phase. An ideal model-based agent achieves a LoCA regret of 0.
Datasets and Baselines
The researchers evaluated their metric using two primary environments:
- Gridworld (25x4): A tabular environment used to validate the metric against classic algorithms like Sarsa($\lambda$) and Model-Based Value Iteration (MB-VI).
- Modified Mountain Car: A variation of the classic benchmark. They added a second terminal state (T2) and a "one-way passage" to ensure the agent could only learn about the local change at T1 without seeing the rest of the map.
They tested several prominent algorithms, including Sarsa($\lambda$), MuZero (a state-of-the-art model-based learner), and PlaNet.
Key Findings
Zero Regret is Possible for True Planning
In the tabular Gridworld, the model-based MB-VI achieved 0.00 LoCA regret, instantly adapting to the new optimal path. In contrast, the model-free Sarsa($\lambda$) had a significant regret, taking time to unlearn its previous habits. This provides a clear "North Star" for deep RL methods to aim for.
The "MuZero Gap"
While MuZero is a powerhouse in standard benchmarks, it does not achieve 0 LoCA regret. On the Mountain Car task, MuZero achieved a LoCA regret between 2.24 and 5.93 (depending on hyperparameters). While this was significantly better than Sarsa (regret of 7.30), it confirms that MuZero still exhibits some "habitual" delay in adaptation and its "model-basedness" isn't yet perfect.
Representation vs. Model
The paper demonstrates that a "good representation" can make a model-free agent look efficient, but it won't lower its LoCA regret. LoCA successfully disentangles representation learning from true model-based planning.
The Danger of On-Policy Models
The authors found that models conditioned on the current policy (on-policy models) can actually hinder adaptation. If a model only learns about transitions it expects to take, it struggles to plan for "what-if" scenarios when the environment changes.
Limitations and Future Work
- Environment Constraints: The LoCA setup requires specific environment features, such as two terminal states and a way to restrict the agent to a local area. Not all RL benchmarks can be easily adapted to this format.
- Stochasticity: The current study focused largely on deterministic environments. "Expectation models" (like MuZero's) might struggle with compounding errors in stochastic settings, which is a prime candidate for future research.
- Scaling: Future work is needed to see how LoCA regret behaves in high-dimensional visual environments like Atari or 3D physics simulations.
Summary updated