Overcoming Compounding Error in Model-Based Planning

949 concepts586 papers3 proposals

In model-based reinforcement learning (MBRL), an agent learns a "world model"—an internal simulator that predicts the next state of the environment given the current state and action. In theory, a perfect world model allows an agent to imagine infinite futures, planning optimal sequences of actions without ever risking failure in the real world.

In practice, however, these internal simulations frequently collapse over long horizons. This failure is driven by compounding model error. When an agent plans, it unrolls its world model autoregressively: it takes its own predicted next state and feeds it back into the model as the input for the subsequent step. If the model's prediction contains even a tiny error, the next prediction is evaluated on an out-of-distribution state. This statistical discrepancy amplifies geometrically with each step, causing the imagined trajectory to rapidly diverge into unrealistic "hallucinations." As formalized by Ruosong Wang in What are the Statistical Limits of Offline RL with Linear Function Approximation?, this error amplification creates an exponential sample complexity wall over extended planning horizons.

To make world models reliable, researchers have developed several distinct strategies to mitigate, bypass, or fundamentally restructure how future trajectories are simulated.


Changing the Direction and Target of Prediction

Traditional world models attempt forward state prediction, mapping $s_t, a_t \to s_{t+1}$. In high-dimensional spaces (like raw pixels), this forces the model to waste capacity tracking task-irrelevant background noise, which accelerates error accumulation.

One intuitive solution is to change what the model predicts. Keiran Paster et al. introduce GLAMOR in Planning from Pixels using Inverse Dynamics Models, shifting the focus from forward pixel prediction to inverse dynamics. By predicting the sequence of actions required to transition between states rather than predicting the future states themselves, the model ignores ambient environmental noise and eliminates forward compounding drift.

Alternatively, agents can change the direction of planning. In Regression Planning Networks, Danfei Xu et al. design a framework that plans backward from a desired goal state to the current observation. Reversing the planning direction allows the system to reason about preconditions and subgoal dependencies, avoiding the sequential forward rollouts that cause rapid trajectory divergence.

To bypass state-to-state transitions entirely, some approaches model long-term outcomes implicitly. Tyler Han develops DiSPOs in Transferable Reinforcement Learning via Generalized Occupancy Models, which leverages successor features and diffusion to learn a global distribution over all possible long-term outcomes from an offline dataset. Because the model evaluates long-horizon consequences in a single step rather than compounding errors over a sequence of transitions, it remains robust across extended horizons.


Temporal Abstraction and Hierarchical Skills

Compounding error is a function of the number of sequential transitions the model computes. If an agent operates at a high control frequency (e.g., predicting steering angles every millisecond), a long-distance plan requires thousands of steps, making simulation collapse inevitable.

Temporal abstraction shortens the effective planning horizon by grouping low-level actions into macro-actions or "skills." In Boosting Offline Reinforcement Learning for Autonomous Driving with Hierarchical Latent Skills, Hang Zhao 0021 et al. propose a hierarchical framework (HsO-VP) that separates discrete driving choices (like turning) from continuous execution. By planning over these extended latent skills rather than high-frequency control steps, the agent drastically reduces the number of autoregressive transitions, mitigating error accumulation over long distances.

High-Frequency Planning (Error compounds at every micro-step):
[s0] -> (a0) -> [s1] -> (a1) -> [s2] -> (a2) -> [s3] -> ... -> [sN] (High Divergence)

Hierarchical Skill Planning (Fewer transitions, stable horizon):
[s0] ----------> (Skill: Turn Left) ----------> [s_mid] ----------> (Skill: Merge) ----------> [s_goal]

This concept is extended to state representations by Xianghua Zeng et al. in Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning. The authors use graph-based structural information to adaptively cluster states into a multi-scale hierarchy. This structural abstraction ensures that variance and prediction errors do not accumulate destructively, enabling stable planning in sparse-reward environments.


Uncertainty-Aware and Masked Imagination

If a world model must make forward predictions, the agent should at least recognize when its imagination has become unreliable. By quantifying prediction uncertainty, an agent can discount or truncate trajectories before they diverge into nonsense.

In Trust the Model When It Is Confident: Masked Model-based Actor-Critic, Feiyang Pan et al. introduce an uncertainty-aware masking mechanism (M2AC). The algorithm monitors the variance of its predictions and dynamically filters out unreliable simulated trajectories, allowing the agent to safely utilize longer planning horizons where the model is confident while ignoring highly noisy rollouts. A similar philosophy underpins the work of Cornelius Weber et al., whose learning-adaptive imagination framework restricts robotic planning exclusively to regions of the state space where the internal model has proven historically accurate.

When dealing with complex physical interactions like contact-rich manipulation, even minor initial errors can render final state predictions useless. To address this, Alberto Rodriguez 0003 and Nima Fazeli develop a hybrid framework combining traditional physics engines with data-driven residual learners. By explicitly propagating stochastic uncertainty through the transition dynamics, their model prevents minor contact inaccuracies from blowing up over extended horizons.


Non-Autoregressive Trajectory Generation

The root cause of compounding error is the autoregressive nature of step-by-step prediction. A radical alternative is to generate the entire multi-step future trajectory simultaneously, optimizing the sequence globally rather than sequentially.

Marc Rigter et al. implement this via PolyGRAD in World Models via Policy-Guided Trajectory Diffusion. Instead of predicting $s_{t+1}$ from $s_t$, PolyGRAD takes a block of random noise representing an entire multi-step horizon and refines it into a coherent sequence of states and actions using a diffusion process guided by the agent's policy. Because the entire trajectory is optimized globally in a single pass, the model completely bypasses step-by-step error accumulation.

Autoregressive Generation (Sequential dependency breeds drift):
Step 1: s0 -> [Model] -> s1_err
Step 2: s1_err -> [Model] -> s2_double_err
Step 3: s2_double_err -> [Model] -> s3_hallucination

Diffusion Trajectory Generation (Global refinement eliminates sequential drift):
Pass 1: [ Noise , Noise , Noise , Noise ]
Pass 2: [ Rough , Rough , Rough , Rough ]
Pass 3: [ Clean , Clean , Clean , Clean ] (Coherent multi-step plan)

Taking non-parametric modeling to its logical conclusion, Federico Malato and Ville Hautamaki eliminate learned transition functions entirely in Zero-shot World Models via Search in Memory. Instead of using a neural network to parameterize dynamics, their system performs a similarity search over a persistent memory of real past experiences to construct future rollouts, ensuring that the simulated states are always grounded in actual historical reality rather than unconstrained network hallucinations.


Robust Objectives and State Representations

Finally, compounding error can be mitigated by changing how world models are trained and represented. Standard models minimize one-step-ahead prediction errors (such as Mean Squared Error), which does not penalize long-term drift.

Andreas Doerr et al. address this training mismatch by introducing Multi-Step Gaussian Processes (MSGP) in Optimizing Long-term Predictions for Model-based Policy Search. Instead of optimizing for single-step accuracy, MSGP trains the model to maximize the likelihood of entire multi-step rollouts, explicitly penalizing the model for long-term trajectory divergence.

The mathematical structure of the latent space also dictates how errors propagate. Continuous latent spaces are susceptible to small numerical drifts that snowball over time. To counter this, Yi Zhao proposes DC-MPC in Discrete Codebook World Models for Continuous Control. By quantizing environment states into discrete codes from a fixed codebook and training the dynamics via classification rather than regression, the discrete boundaries prevent small continuous errors from accumulating, stabilizing long-term planning for complex continuous control tasks.

To directly combat the distribution shift that occurs between real-world data and simulated rollouts, Jian Shen et al. introduce AMPO in Model-based Policy Optimization with Unsupervised Model Adaptation. By using unsupervised domain adaptation to explicitly align the feature distributions of real and simulated data, the authors minimize the distribution mismatch, significantly reducing compounding errors during multi-step policy optimization.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_compounding_model_error_25662aaa,
  title = {compounding model error},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compounding-model-error/rdr_25662aaa48d24f4e9e03e7005be02f60},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_compounding_model_error_25662aaa_v1,
  title = {compounding model error},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/compounding-model-error/rdr_25662aaa48d24f4e9e03e7005be02f60/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake