A video world model is a video generator that is told what an agent does, for example "move the gripper left", and imagines what happens next. Robots and game-playing agents can then plan or learn inside these imagined futures. The flaw we target is simple. In most such models the action is broadcast to every spatial position of the frame, so pressing "left" can quietly change things the agent cannot touch: a car drifting on a highway, a ball's trajectory before the paddle reaches it, clutter on a table. Brains face the same problem and solve it with an efference copy, an internal duplicate of each motor command. An internal predictor, called a forward model, uses the copy to forecast the sensory consequences of the movement, so the brain can separate self-caused signals from signals caused by the world.
We propose to install this mechanism inside a frozen video diffusion model (a generator that turns random noise into video by repeated denoising) at sampling time, with no retraining. A small forward model predicts, from the action and the robot's state, where the agent's body will be in the next frame. That prediction then controls which image tokens may read the action signal directly. Tokens on the predicted body receive the real action. All other tokens receive a "null action", meaning "do nothing". Those tokens can still change, but only by attending to body tokens through the model's ordinary self-attention. In plain terms, the action is allowed to affect the world only through the agent's body. That is the physical principle of "no action at a distance", enforced by routing attention rather than by editing pixels. It should remove action bleed while keeping real contact effects, such as a pushed block moving or a ball bouncing off the paddle.
Core Claim
Action-conditioned video world models show a spatial analogue of the attribute bleeding that attention-steering methods fix in text-to-image models. We claim this bleed can be removed at inference by gating the action-conditioning pathway token by token. The gate comes from a graded efference-copy prediction of where the body will be, and interactions are preserved because the self-attention pathway is left intact. Write $e(a)$ for the model's embedding of action $a$ and $e(\varnothing)$ for the embedding of the null action. Write $m_i$ for the forward model's predicted probability that token $i$ lies on the agent's body in the frame being generated, and $\sigma_i$ for that model's positional uncertainty. The conditioning seen by token $i$ is
$$c_i = g_i\, e(a) + (1-g_i)\, e(\varnothing), \qquad g_i = \big(m * k_{\sigma_i}\big)_i ,$$
where $k_{\sigma_i}$ is a blur kernel whose width grows with the uncertainty. When the forward model is unsure where the arm will be, the gate widens and suppression relaxes.
For models that condition by scale-and-shift, $c_i$ replaces the shared action modulation. For models that condition through cross-attention to action tokens, $g_i$ scales token $i$'s attention scores to those tokens.
The design is "graded, not binary" because of the finding that neural suppression of self-produced speech scales with how precisely feedback matches the prediction (Christoffels et al.). Existing diffusion steering, in contrast, uses hard inside/outside masks.
Relation to Prior Work
Three papers motivate the problem and none of them solves it at inference. Policy-Shaped Prediction shows that reconstruction-based world models such as DreamerV3 spend capacity on predictable but irrelevant dynamics. It adds an efference-copy-style adversarial head that predicts actions from the encoding, but only as a training loss on a small latent model. Pre-Trained Video Generative Models as World Simulators turns Open-Sora into an action-conditioned simulator by training scale-and-shift adapters and a motion-weighted loss. That loss amplifies all motion without distinguishing self-caused motion from world-caused motion, and the adapter is exactly the broadcast pathway we gate. MultiModal Action Conditioned Video Simulation binds rich actuation signals to video through channel-wise cross-attention, again learned end-to-end and without any suppression. We reuse the first paper's intuition, the second paper's architecture as our test model, and neither paper's training step.
The method draws on two other lines. The first is counterfactual reasoning: Disentangling Reafferent Effects by Doing Nothing defines self-caused change as the difference between a forward model run with the real action and the same model run with a null action. Its demonstrations are single-step and limited to Cartpole, Freeway and a toy ape. We use the null action inside every step of a multi-step diffusion sampler, and per token rather than per frame.
The second is training-free attention steering. DenseDiffusion boosts text-to-region cross-attention inside layout masks and suppresses it outside. VideoGrain suppresses inter-region interference in video self-attention. MotionFlow steers motion by optimizing latents against attention losses. All three take their masks from users or reference videos and their steering signal from text. Ours takes its mask from a motor command and its signal from an action embedding. It also deliberately leaves self-attention unmasked, because self-attention is the channel that contact must travel through.
Two papers constrain the design. MotionFlow argues that overwriting attention maps pushes models into unnatural states. We answer this in two ways. First, we never overwrite a map; we substitute a conditioning vector the model has seen in training (the null action) on a subset of tokens. Second, DenseDiffusion reports that value-range-adapted direct modulation works without artifacts. MAGS shows in language models that state-dependent, thresholded intervention preserves fluency where fixed additive steering vectors degrade it. This supports a token-dependent gate over a global nudge.
The biological literature, including the forward-model loop (McNamee et al.) and anticipatory blink suppression that starts roughly 100 ms before the eyelid closes (Willett et al.), supplies two design choices. The gate should be computed from a prediction of the frame being generated, not from the previous frame. And the gate is part of the forward pass, not a post-hoc filter.
Research Program
We first build the test model. We reproduce a DWS-style action-conditioned video transformer on BAIR robot pushing, on rendered LIBERO manipulation episodes, and on Atari Freeway and Breakout, then freeze it. Freeway is the clean case for pure exafference, meaning world-caused change: cars never depend on the chicken. Breakout is the stress test for contact: the ball is outside the body until the paddle hits it.
Next we build the forward models. For LIBERO the efference prediction is exact: we render the robot's segmentation from the commanded joint states. For Atari it is the sprite position obtained from the emulator's RAM. For BAIR we train a small convolutional predictor from (frame, action) to an arm mask, supervised with Segment Anything masks of the arm in the same way PSP uses SAM. The predictor's error is deliberately left nonzero so that it drives $\sigma_i$.
We then study three questions. First, we characterize bleed in unsteered models: how much non-contact regions vary across actions from the same starting frame, and at which layers and denoising steps the action signal enters those regions. Second, we sweep where the gate is applied: all timesteps versus only early ones, since DenseDiffusion intervenes only at $t \in [0.7, 1.0]$. Third, we compare graded gates against hard gates, and gates driven by the true body mask against gates driven by the learned forward model. The last comparison measures how much forward-model error the method tolerates.
A secondary study uses Reafferent DMC, whose backgrounds change as a function of the agent's own actions and so violate our locality assumption on purpose. There we expect the steered model to diverge from ground-truth pixels. The question is whether agents trained on its imagined rollouts are nonetheless more robust, which is PSP's thesis moved to inference time.
Baselines
The naive reference point is the frozen, unsteered action-conditioned model sampled normally. All other methods use the same checkpoint, sampler, step count and random seeds.
The strongest existing method is the fine-tuned DWS simulator itself (He et al.). On BAIR it reports reducing FVD (Fréchet Video Distance, a measure of how far generated videos are from real ones) by 11.7% and LPIPS (a perceptual image difference) by 27.9% relative to baseline fine-tuning. On Breakout it reports a 7x agent performance gain over DreamerV3. These are relative figures, so we will reproduce the absolute numbers ourselves. To beat it, our steered model must not lose meaningfully on FVD or LPIPS, and must clearly reduce bleed and improve downstream agent reward.
We also train a strengthened version of DWS: we continue fine-tuning it with an explicit invariance loss that penalizes differences between action and null-action predictions outside the ground-truth body mask. This is given the same compute budget our method spends on its forward model plus sampling overhead, and its hyperparameters are tuned on the same validation split. It is the fairest "just train it" competitor.
For LIBERO-LONG we plug steered and unsteered imagination into the planning loop of LaDi-WM, which reports 90.7% average success. Beating it means a statistically reliable gain from steering alone, with the policy held fixed.
Three baselines isolate the key ingredient. The first is null-branch latent compositing: run the action and null-action branches in parallel and blend their denoised latents by $g_i$ at each step, as in inpainting-style blending. It tests whether routing through attention matters or whether blending outputs suffices. The second is a hard binary gate with the same mask, in the style of DenseDiffusion. It tests whether grading matters; DenseDiffusion's reported mask-grounding IoU of 34.99 versus 23.76 for SD-Pww shows that hard masking is itself strong in its own setting. The third is a static action-steering vector subtracted from non-body tokens. It tests whether the dynamic, prediction-driven gate is needed.
For Reafferent DMC, the reference is PSP's reported 2x improvement over DreamerV3 and DreamerPro.
Why not just…?
Why not just fine-tune the world model to be invariant to actions outside the body? This is a real threat, and it is the strengthened-DWS baseline above. Its costs are that it needs body masks for every training frame, it modifies a foundation model whose generality the adapter literature tries to preserve (Self-Guidance, DenseDiffusion), and it must be redone for each new body or camera. If it beats inference-time gating at matched compute on both bleed and contact fidelity, we will report that. The remaining contribution would then be the diagnosis of bleed and the gate as a cheap zero-shot fallback. We expect the two to be close on in-distribution data and the gate to win when the forward model changes at test time, for example a different gripper, because only the gate can adopt the new body without retraining.
Why not just generate a null-action rollout and paste in the body region from the action rollout? Compositing in latent or pixel space is exactly the counterfactual difference proposed by Wilkins and Stathis. It will remove bleed, perhaps as well as our method on Freeway. Its weakness is contact. Outside the mask it copies the do-nothing future, so the Breakout ball keeps flying through the paddle and the BAIR block never moves. Our method leaves self-attention from world tokens to body tokens active, so consequences can propagate. The contact-fidelity metric below decides this directly. If compositing matches us on contact, the attention-level design is unnecessary and we will say so.
Why not just subtract a fixed "action direction" from background activations? MAGS shows in language models that fixed additive steering vectors over-steer and raise perplexity, while intervening only on states that project onto a measured error direction does not. Our static-vector ablation checks whether the same holds for video. We expect the action's effect on background tokens to depend on the scene and on the denoising step, which a single fixed vector cannot capture.
Proposed Experiments
The primary metric is counterfactual exafference error. From a shared starting frame we sample rollouts under several actions and compare the non-contact regions against re-simulated ground truth. Ground truth comes from LIBERO and Atari, where we can replay the emulator, and from held-out BAIR trajectories. Non-contact regions are defined by ground-truth segmentation minus the body and minus anything the body touches. The companion metric is contact fidelity: the tracked displacement error of objects the body actually touches, and ball-bounce correctness in Breakout. We found no reported result on Lacuna for either metric on any of these benchmarks, so the unsteered model sets the reference. We also report BAIR FVD and LPIPS, sampling latency relative to the unsteered model, and downstream task performance: Atari reward from agents trained in imagination, and LIBERO-LONG planning success.
The decisive checks are these. First, bleed should drop sharply on Freeway while contact fidelity on Breakout and BAIR stays at the unsteered level; a drop in contact fidelity would falsify the "no action at a distance" routing argument. Second, graded gating should outperform hard gating specifically when the forward model is noisy, which is the BAIR learned-mask condition and the perturbed-mask condition. Third, the gain in agent reward should track the reduction in bleed across ablations. If bleed falls and reward does not move, bleed was not what limited planning. Fourth, on Reafferent DMC, the locality violation should show up as higher pixel error. If agents still improve, that suggests inference-time sensory suppression recovers part of what PSP achieves only through training.
References
- The Sensory Consequences of Speaking: Parametric Neural Cancellation during Speech in Auditory Cortex
- Policy-shaped prediction: avoiding distractions in model-based reinforcement learning
- Pre-Trained Video Generative Models as World Simulators
- MultiModal Action Conditioned Video Simulation
- Disentangling Reafferent Effects by Doing Nothing
- Dense Text-to-Image Generation with Attention Modulation
- VideoGrain: Modulating Space-Time Attention for Multi-Grained Video Editing
- MotionFlow: Attention-Driven Motion Transfer in Video Diffusion Models
- Manifold-Guided Attention Steering
- Internal Models in Biological Control
- The perceptual consequences and neurophysiology of eye blinks
- LaDi-WM: A Latent Diffusion-Based World Model for Predictive Manipulation
- Diffusion Self-Guidance for Controllable Image Generation