How Uncertain Is My Action? Self-Correcting Visual Planning with Frozen World Models

Siqing Li, Luming Yang, Yijia Jiao, Junze Deng, Alper YilmazarXiv 2026 · arXiv

TL;DR

ACTUS (Action Uncertainty Self-correction) allows robots to detect and fix unknown actuation errors—like a slipped joint or a miscalibrated motor—using a "frozen" world model. By running the model in reverse to infer what action actually happened versus what was commanded, the system can self-correct its plans in real-time without any retraining or weight updates.

What Problem Does This Paper Address?

Modern visual planners rely on World Models (WMs)—neural networks trained to predict how the world will look after a robot takes a specific action. Usually, these models are "frozen" after training. At deployment, however, reality often diverges from the lab: a motor might lose power, a joint might slip, or an axis might be misaligned.

We call this an actuation shift. Because the world model is frozen, it has no idea that its commands aren't being executed perfectly. It continues to plan based on "imagined" outcomes that no longer match reality, leading to catastrophic task failure. While some researchers try to "fine-tune" the model on the fly, this is often slow, unstable, and requires significant data. This paper asks: Can we use the frozen model itself to figure out what's going wrong and fix it?

Methods

The authors introduce ACTUS, a framework that treats a frozen world model as a "counterfactual simulator." Instead of just using it to predict the future (forward pass), they use it to explain the past (backward pass).

  1. Inverse Counterfactual Formulation: They model actuation shifts as geometric transformations (Lie groups) like rotations or gains.
  2. The Shift Score: Instead of looking at the total prediction error (residual energy), which is often "noisy" due to the model's own inherent inaccuracies, ACTUS calculates a shift score. This score projects the prediction error onto the specific directions that a physical shift would cause, using Jacobian-vector products (JVPs).
  3. Rao Test (The Alarm): ACTUS uses a statistical test to monitor the shift score. If the score deviates significantly from what's expected during normal operation, an "alarm" fires.
  4. Estimation & Compensation: Once the alarm sounds, the system uses Gauss-Newton optimization to estimate the exact magnitude of the shift. It then "re-plans" by applying the inverse of that shift to all future commands, effectively "steering into the skid" to cancel out the error.
Figure 3. Overview of ACTUS. The frozen model plans as usual and the actuator applies the unknown shift T(θ) (red). The residual and one JVP per generator give (st, Ft), pooled into (S, F) for alarm, the estimate ˆδ; after the first alarm the planner re-plans through T(ˆδ)−1.

Datasets

The researchers evaluated ACTUS across several challenging visual planning benchmarks:

  • PushT: A standard task where a robot must push a T-shaped block to a target.
  • TwoRoom: A navigation task requiring planning through doorways.
  • OGBench-Cube: A difficult spatial manipulation task involving moving a cube in 3D.
  • Real-world SO-101 Robotic Arm: Testing on physical hardware with both software-injected shifts and natural mechanical offsets.

They tested these using two different state-of-the-art world models: LeWM (Latent World Model) and DINO-WM.

Key Insights and Ideas

  • Frozen Models are Better Than We Thought: A key "aha moment" is that a frozen model already "knows" enough about its actions to diagnose its own failures. You don't need to change the neural network's weights; you just need to change how you interpret its outputs.
  • Score vs. Energy: The paper proves that the standard way of measuring error (residual energy) is a poor detector for small shifts because it's confounded by the model's own bias. The shift score is far more sensitive because it ignores general model noise and looks only for "shift-shaped" errors.
Figure 2. Energy against score (Prop. 1). (a) a shift moves the residual rt only inside the span of Jt (blue), the model error εt in all D directions. (b,d) change of the whitened energy E (grey) and of the score (blue) on PushT along two generator axes. (c) AUROC against the Fisher norm, 124 PushT shifts; line: Prop. 2.
  • Calibration Matters: By observing how the model behaves during "nominal" (error-free) runs, ACTUS can set a precise threshold for its alarm, ensuring it doesn't fire unnecessarily while remaining highly sensitive to real problems.

Key Results

  • Near-Oracle Performance: On the PushT task, ACTUS raised the success rate from 0.650 to 0.862, virtually matching an "oracle" that was told the exact shift by a human.
  • Superior to Adaptation: ACTUS outperformed five common "test-time adaptation" methods (which update model weights), proving that modeling the actuator is often more effective than trying to fix the brain.
  • Real-World Impact: On a real robotic arm, ACTUS detected a $10^\circ$ base-joint offset that the robot's own encoders couldn't see. It reduced the reaching error from 36mm down to just 3-5mm, which is the limit of the robot's mechanical repeatability.
  • Predictable Detection: The authors showed that detection performance (AUROC) follows a universal theoretical law, allowing engineers to predict exactly how many steps it will take to detect a shift of a certain size.
Figure 4. Shift detection. (a) AUROC (n = 20) by Fisher norm of the shift. (b) Measured against predicted GLR delay (Eq. (10)).

Limitations and Future Work

  • Geometric Assumptions: Currently, ACTUS assumes shifts can be described by Lie-group transformations (like rotation, scale, or shear). It might struggle with highly non-linear or complex "black box" actuation failures.
  • Nominal Data Requirement: The system requires a small amount of "clean" data to calibrate its noise floor before it can reliably detect shifts.
  • Future Directions: The authors suggest that this "inverse" use of world models could become a standard part of the robotics stack, acting as a real-time filter to verify action validity and improve safety in unpredictable environments.

Summary updated

Abstract

A frozen latent world model combined with sampling-based model predictive control is a visual planner. At deployment, the executed action can depart from the commanded one, and the frozen model has no way to notice. We show that the model can be run in reverse: for any hypothesized actuation shift, it predicts what it should have observed under that shift, and comparing this prediction with what was actually observed quantifies the shift and lets the planner correct itself. Unlike the residual energy, which mixes the shift with the model's own error, the score keeps only the part of the residual that a shift can produce. This shift score detects the shift with a locally optimal test, estimates it with Cramer-Rao error bars, names the dominant generator, and compensates the planner once an alarm fires. On PushT, TwoRoom and OGBench-Cube with a frozen latent world model, measured detection follows the prediction to within 0.05 AUROC. Self-correction raises success from 0.650 to 0.862 on PushT and from 0.378 to 0.432 on the harder Cube task, levels with an oracle that is told the true shift at the same block, better than five common test-time adaptation methods. On a real robotic arm, the score identifies a base-joint offset that no joint encoder registers and reduces the reach error from 36 mm to 3-5 mm, and under shifts drawn at random and sealed before the session, estimates recover 92-94% of the error reduction achieved with the true shift. The broader implication is that a frozen world model can act as a filter by predicting the future, testing action-interface uncertainty and validity, and finally correcting failures.
BibTeX
@misc{li2026how,
  author = {Siqing Li and Luming Yang and Yijia Jiao and Junze Deng and Alper Yilmaz},
  title = {{How Uncertain Is My Action? Self-Correcting Visual Planning with Frozen World Models}},
  year = {2026},
  eprint = {2611.23620},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23620},
}
Figure