Abstract
Model-based reinforcement learning (MBRL) often fails because small prediction errors compound over long planning horizons. This paper provides a theoretical foundation for "self-correcting" models—models trained to recover from their own past mistakes—and introduces H-DAgger-MC, an algorithm that offers performance guarantees even when the model is fundamentally incapable of perfectly representing the environment.
Introduction
In MBRL, an agent learns a model of the world and "imagines" future outcomes to make decisions. The primary advantage of this approach is sample efficiency: learning from a model is theoretically faster than trial-and-error in the real world. However, in practice, MBRL often performs worse than model-free methods in complex tasks due to compounding error.
If a model has a tiny error at step one, it might predict a state the agent has never seen before. Because the model wasn't trained on that "hallucinated" state, its subsequent predictions become increasingly inaccurate. Standard training focuses on one-step accuracy, which is a poor proxy for how well a model performs when its own predictions are fed back into it as inputs over long horizons.
Methodology: Hallucinated Replay
To address compounding errors, the authors build on a technique called Hallucinated Replay. Instead of only training the model to predict the next state $s_{t+1}$ from a real state $s_t$, the model is trained to predict the correct environment state $s_{t+1}$ even when its input is a flawed, "hallucinated" state $z_t$ generated by the model itself.
This "self-correcting" property ensures that even if the model makes a small mistake, it knows how to "get back on track" in the next step. The paper derives new theoretical bounds showing that "hallucinated one-step error" is a much tighter predictor of planning performance than standard one-step error in deterministic environments.
The H-DAgger-MC Algorithm
The proposed H-DAgger-MC algorithm combines Hallucinated Replay with DAgger (Data Aggregator). A major insight is that using one single model for every step of a hallucinated rollout can be unstable; changing parameters to fix an error at a late step might ruin accuracy at an earlier step.
H-DAgger-MC "unrolls" the model, using a sequence of models where each is responsible for one specific step of a rollout. This prevents feedback loops and ensures training stability. The method utilizes a simple One-Ply Monte Carlo planner that evaluates actions by taking many $T$-step rollouts in the learned model.
Experimental Results
The primary evaluation is conducted on the "Shooter" domain, where an agent moves a spaceship to shoot targets. In the "Moving Bullseyes" variant, targets move back and forth. Since the agent's model is Markovian (seeing only the current frame), it cannot perceive velocity, making the model class inherently flawed.
As shown in the experimental results, standard MBRL and DAgger-MC fail catastrophically in the Moving Bullseyes task because they cannot handle the compounding errors caused by missing velocity information. In contrast, H-DAgger-MC achieves near-optimal performance (average return of ~7), while standard methods remain near the level of a random policy.
Discussion and Limitations
While H-DAgger-MC shows significant promise, several limitations remain. The strongest theoretical results rely on deterministic environments; extending this to stochastic real-world robotics is an open challenge, as hallucinated training can be harmful if the model tries to "correct" for legitimate random outcomes. Additionally, "unrolling" the model increases computational overhead, and future work is needed to generalize the theory to state-dependent rollout policies.
Summary updated