Locally Accurate World Models for Reinforcement Learning

This direction is split into other directions: local dynamics modeling. Its last version stays readable and citable here.

Version 1 of 131 papers49 conceptsfrozen 2026-10-04

In the pursuit of building intelligent agents, researchers often assume that a "world model"—a simulator of the environment—must be globally accurate to be useful. The intuition is straightforward: if an agent can perfectly predict the outcome of any action in any state, it can plan the optimal path to success.

However, in complex, high-dimensional environments, achieving global accuracy is often mathematically intractable and computationally wasteful. A growing body of research suggests a more efficient alternative: local fidelity. Instead of trying to master the entire universe, an agent only needs a model that is accurate in the specific regions it is currently visiting.

The Fallacy of Global Accuracy

Traditional model-based reinforcement learning (RL) often fails because of the "Reality Gap." When a model is trained to minimize average error across the entire state space, it may achieve low global error while remaining disastrously inaccurate in the narrow corridors of the state space that actually matter for the task.

As Muxi Tao, Jiangtao Wen, and Yuxing Han argue in Prioritized Model Experience Replay, high global accuracy can mask local failures. If the model is wrong about the specific transitions the current policy is attempting, the agent will follow "imaginary" gradients into instability.

Local Derivatives over Global Fidelity

The insight that models don't need to be perfect to be useful dates back to early work by Pieter Abbeel, Morgan Quigley, and Andrew Y. Ng. In Using inaccurate models in reinforcement learning, they demonstrated that even a crude simulator can facilitate learning as long as its local derivatives are correct.

If the model correctly predicts that "moving the joint slightly left increases velocity," the agent can improve its policy, even if the model is wildly wrong about the robot's absolute position. This theme is echoed in recent work by Nan Fang and colleagues in Efficient Model-Based Reinforcement Learning for Robot Control via Online Learning, where they show that locally correct gradient directions allow robots to learn directly in the real world without a high-fidelity simulator.

Aligning the Model with the Policy

If the goal is local accuracy, the next question is: where should the model be accurate? The answer is the "on-policy" distribution—the states and actions the agent is currently exploring.

In the context of human feedback, Xiao Hu and Jianxiong Li identify a "Query-Policy Misalignment" in Query-Policy Misalignment in Preference-Based Reinforcement Learning. They argue that asking humans to label behaviors the agent isn't currently performing is a waste of effort. By focusing reward model accuracy on the agent's immediate vicinity, they achieve massive gains in feedback efficiency.

Similarly, Xiyao Wang and Ruonan Jia propose PDML in Live in the Moment: Learning Dynamics Model Adapted to Evolving Policy. Their approach dynamically reweights historical data to ensure the dynamics model prioritizes the current policy's distribution. While this sacrifices the model's ability to generalize to distant regions, it creates a "highly accurate local simulator" that accelerates task-specific progress.

Robustness to Misspecification

A theoretical pillar of this research direction is that agents can remain optimal even when their internal models are "misspecified" (inherently incapable of representing the true environment).

Yunfan Li provides a unified framework for this in Least Square Value Iteration is Robust Under Locally Bounded Misspecification Error. The core proof shows that as long as the model is accurate on average under the distribution of states induced by the agent's policy, the agent can achieve near-optimal performance. This relaxes the restrictive requirement for global accuracy, acknowledging that function approximators like neural networks only need to be precise in relevant regions.

This is further extended to the offline setting by Kefan Dong and Emma Brunskill in Model-Based Offline Reinforcement Learning with Local Misspecification. They demonstrate that prioritizing local fit over global consistency allows for tighter performance bounds, preventing the policy from being penalized for model errors in irrelevant parts of the environment.

Safe Neighborhoods and Trust Regions

Finally, this direction provides a path toward Safe Policy Improvement (SPI). If we know our model is only locally accurate, we should only update our policy within a "neighborhood" where we trust the model's predictions.

Florent Delgrange, Raphaël Avalos, and Willem Röpke formalize this in Deep SPI: Safe Policy Improvement via World Models. By constraining updates to a specific neighborhood and using local losses, they ensure monotonic improvement. This "trust the model where it trusts itself" philosophy—also explored by Bernd Frauenknecht in MACURA—replaces rigid simulation schedules with adaptive rollouts based on local uncertainty.

Summary

The shift from global to local accuracy represents a move toward more pragmatic AI. By acknowledging that a model is a "local specialist" rather than a "global oracle"—a phenomenon Jinke He observed even in powerful systems like MuZero—researchers are building RL agents that are more robust, sample-efficient, and capable of operating in the messy, unpredictable real world.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_locally_accurate_world_models_for_reinfo_90d5057a,
  title = {Locally Accurate World Models for Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/locally-accurate-world-models-for-reinforcement-learning/rdr_90d5057a572841849a25fd7cfb35ee72},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_locally_accurate_world_models_for_reinfo_90d5057a_v1,
  title = {Locally Accurate World Models for Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/locally-accurate-world-models-for-reinforcement-learning/rdr_90d5057a572841849a25fd7cfb35ee72/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}