In high-dimensional robotics and reinforcement learning, the "curse of dimensionality" suggests that learning a perfect global model of the environment is nearly impossible. A robot attempting to master every possible interaction in its state space would require an astronomical amount of data. However, a recurring insight across decades of research is that global accuracy is often unnecessary. Instead, agents can achieve expert-level performance by focusing their representational capacity on localized dynamics models—simulators that are highly accurate only within the specific regions, trajectories, or distributions relevant to the task at hand.
The Efficiency of Being "Right Here"
The fundamental motivation for localized modeling is sample efficiency. As Schaal and Atkeson argued, attempting to learn an entire high-dimensional state space is computationally intensive and involves redundant information. By using techniques like Locally Weighted Regression, they demonstrated that a system could master complex tasks by prioritizing data collection in the specific regions necessary for a task trajectory.
This "trajectory-centric" view allows robots to handle extreme operating regimes that would baffle a general-purpose model. For instance, Coates et al. showed that for complex, non-linear helicopter maneuvers, modeling dynamics locally along an intended path is far more effective than building a single global physics model. By focusing on the "local" nuances of a specific flight path, the system can surpass the performance of the human teacher who provided the initial demonstrations.
Modeling the Policy's Neighborhood
A dynamics model does not exist in a vacuum; it serves the agent's current policy. Several researchers have proposed that a model only needs to be accurate within the state-action distribution that the current policy actually visits.
- Distributional Weighting: Wang and Jia introduced Policy-adapted Dynamics Model Learning (PDML), which reweights historical data to focus the model on transitions relevant to the evolving policy. This specialization reduces compounding errors during simulated rollouts.
- Local Regret and Gradients: Preiss et al. proved that for unknown nonlinear systems, an estimator only needs to accurately predict dynamics along the visited trajectory to achieve effective control. Similarly, Fang et al. demonstrated that as long as a model provides a locally correct gradient direction, the policy can continue to improve, even if the model is globally inaccurate.
- Safe Improvement: Delgrange et al. showed that by bounding performance gaps based on local transition and reward errors, agents can safely update their behavior without requiring a perfect simulation of the entire environment.
Mitigating Model Misspecification
One of the greatest risks in model-based RL is "model exploitation," where a policy finds a flaw in a global simulator and achieves a high reward in the simulation that doesn't translate to reality. Localized modeling acts as a defense against this.
Dong et al. argued that prioritizing local fit over global consistency prevents a policy from being penalized for inaccuracies in irrelevant parts of the environment. By accounting for local model misspecification, algorithms can find high-performing policies even when the underlying model class is too simple to represent the entire world. This is echoed by Sun et al., who noted that local dynamics models mitigate model bias by focusing strictly on the current policy's state distribution.
Spatial and Interaction Locality
Locality can also be defined spatially. In manipulation tasks, most actions only affect the immediate vicinity of the robot's grippers. Kohler et al. leveraged this by developing Local Dynamics Models (LDMs) that focus on visual patches centered on interaction points, ignoring static backgrounds to improve resolution and sample efficiency.
In a similar vein, Jin and Posa achieved real-time dexterous manipulation by ignoring thousands of irrelevant contact states, learning simplified hybrid models that only capture the dynamics necessary for the specific task.
Scaling to High Dimensions
For truly high-dimensional systems, such as jet fighters or complex robots, global exploration is not just inefficient—it is dangerous. Smart and Kaelbling proposed "bootstrapping" with a teacher policy to focus data collection on relevant regions from the start.
More recently, Fan and Zhang introduced Lyapunov-Guided Exploration (LYGE), which restricts dynamics modeling to a "trusted tunnel" around known trajectories. This approach allowed them to stabilize an F-16 jet using 95% fewer samples than traditional methods, proving that the path to mastering high-dimensional complexity lies not in seeing everything, but in understanding the local neighborhood of success.
Key Foundational Work
Assessing the Quality of Learned Local Models (1993) — contributed 3 concept elements to this direction.
A Framework for Reinforcement Learning on Real Robots (1998) — contributed 2 concept elements to this direction.