Which Parts of an Imagined Future Does a Robot Policy Actually Use?

Some of the best recent robot policies work by imagining the future. They generate what the camera will see a moment from now and generate the motor commands in the same process, so the imagined frames and the actions come out together. This helps. Prediction with Action (PAD) reports that removing the image-prediction part makes its policy much worse. It also costs a great deal. Generating frames is slow, and most of the work goes into textures, lighting and fine detail. Representation-learning research argues that a model trained to reproduce raw pixels spends its capacity on exactly those details and not on objects or physics (BEiT; Understanding MAE via hierarchical latent variable models). So the field holds two conflicting beliefs: pixel prediction helps policies, and pixel prediction wastes capacity. No one has checked which part of the imagined future the action output depends on.

This proposal checks it directly, using causal interventions of the kind interpretability researchers use: swap or remove parts of the model's internal state and measure what changes downstream. We hypothesize that the actions depend on a small motion-and-contact part of the predicted future, and that appearance matters only as a training-time regularizer, if at all. If that holds, the policy can be trained to predict only the part it uses. We would then make its generation paths straight, so it runs in one or two steps. Straight paths also make a sharper question possible. Methods that compress generation into one step might cut the link between imagined future and action. In that case the policy would ignore its own imagination exactly when imagination is cheapest. We would also test transfer from human video to robots, on the view that human hands and robot grippers differ in appearance but not in the object motion they cause.

Core Claim

In a model that jointly generates future visual states and robot actions, the action output causally depends on a low-dimensional, motion-carrying part of the predicted future state. We can find that part with interventions, and a policy trained to predict only that part will do four things. It will match dense pixel prediction on success rate. It will run at one or two network evaluations. It will transfer from human video better, because the discarded appearance channels are where embodiment-specific information sits.

Three things are new here. The first is using interventions, rather than intuition or probes, to decide what a visual planner should predict. The second is measuring how straightening the generation paths changes the causal link between imagined future and action. The third is treating human-to-robot transfer as a test of which channel carries the transferable signal.

Background and Positioning

Several lines of prior work shape the proposal, and each plays a different role.

The motivating tension comes from two sides. PAD jointly denoises image latents and actions with a diffusion transformer and reaches 72.5% average success on MetaWorld against 57.4% for GR-1. Its ablation says pixel prediction prevents overfitting. The opposing evidence comes from representation learning. V-JEPA predicts in feature space with no pixel decoder and reaches 72.2% on Something-Something-v2 with a frozen backbone. Self-distillation of hidden layers warns in turn that purely abstract targets lose the spatial detail that manipulation needs. Neither side has asked what the downstream action computation consumes.

The generation methods come from the trajectory-straightening literature. A diffusion or flow model turns noise into data by following a path. Flow matching trains a network to predict the velocity along a straight line from noise to data (Nakkiran et al.). Reflow retrains on a model's own noise–output pairs to straighten curved paths further (Guo et al.). InstaFlow shows that this makes one-step image generation workable. WarmPrior straightens robot action policies by starting the path near the recent action history, with the largest gains at one network evaluation. We reuse these techniques. WarmPrior, however, works only on low-dimensional actions and never looks inside its network.

Two constraints limit how far we can push straightening. Reflow can collapse when a model trains on its own outputs (Analyzing and Mitigating Model Collapse in Reflow). InstaFlow reports that one-step models degrade on multi-object composition. Manipulation scenes are exactly that kind of composition.

The interpretability methods come from two studies. Dissociating Decodability and Causal Use removed the directions that linear probes read tree depth from and saw a 0.00% accuracy drop. Knocking out one attention edge caused a 96.7% drop. The lesson is that we cannot use probes to decide what matters. CODEC attributes an output back to internal units and trains sparse autoencoders on those attributions, instead of on activations. The resulting causal modes are sparse, and ablating 2% of channels removes a class selectively. We extend it from scalar class scores to vector-valued action outputs of a generative policy. Both studies were confined to toy transformers and image classifiers, so this is also a test of whether these tools work on continuous generative models.

For cross-embodiment transfer, SuSIE and UniPi use pixels as a shared interface between human and robot video. They pay for it in latency, up to a minute per plan for UniPi, and in hallucinated subgoals. ATM predicts 2D point tracks instead, runs about 500 times faster than UniPi, and reaches 39% on LIBERO-Long against 16% for plain behavior cloning. ATM chose tracks by hand. We ask whether an intervention-guided target recovers something like tracks, or something better.

Research Program

The first stage builds a controlled test bed. We train a single joint flow-matching policy that generates a future visual state together with an action chunk. The visual state can take one of three forms: pretrained image-autoencoder latents (the PAD setting), V-JEPA features, or point tracks produced by an off-the-shelf tracker as in ATM. The transformer, data, and training budget are the same across all three. Each variant is trained with independent noise and again with WarmPrior-style history-anchored starting points, and is evaluated at one, two, and ten network evaluations.

The second stage is a causal audit. We use MetaWorld because simulation lets us render the same episode under different textures, lighting, and object colors while leaving the motion unchanged. That makes clean swap experiments possible. For each trained policy we run the joint generation, then substitute the predicted future visual tokens with tokens from a counterfactual rollout, and measure how much the actions move. The measure is the interchange effect

$$E_{\text{app}} = \mathbb{E}\,\|a(z) - a(z_{\text{app}})\|, \qquad E_{\text{mot}} = \mathbb{E}\,\|a(z) - a(z_{\text{mot}})\|,$$

Here $a(\cdot)$ is the action chunk the policy outputs given predicted future state $z$. The state $z_{\text{app}}$ comes from a rollout with the same motion and a different appearance. The state $z_{\text{mot}}$ comes from a rollout with the same appearance and a different motion.

The hypothesis predicts $E_{\text{mot}} \gg E_{\text{app}}$. We also run the bluntest intervention: drop the predicted visual tokens entirely at inference. That shows whether PAD-style prediction is used at test time or only shapes training. In addition, we run CODEC with each action dimension as the attribution target. The goal is to find the sparse modes in the predicted-state tokens that drive actions, and to check whether the probe-decodable directions (object position, gripper pose) are the same as the causally used ones. Each measurement is repeated across the number of network evaluations, which tests whether one-step generation severs the imagination-to-action link.

The third stage builds the policy the audit points to. We project the visual target onto the causally used modes found in stage two, train a history-anchored flow-matching policy that predicts only that projection plus actions, and apply one round of reflow. We monitor for collapse following the reflow-collapse analysis. The pixel-latent decoder can remain as a training-only auxiliary head. That lets us separately test whether dense appearance prediction helps as a regularizer even if actions never read it.

The fourth stage tests cross-domain transfer. We co-train on Something-Something-v2 human clips, following SuSIE's filtered subset, with action tokens masked as in PAD. We compare co-training that passes human data through the full visual target against co-training restricted to the causally used subspace. We evaluate on CALVIN, and also on BridgeData V2 if robot time allows. The prediction is that restricting human data to the causal subspace keeps or improves the transfer gain, because hand-versus-gripper appearance is embodiment-specific and gets discarded.

Baselines

The naive reference is a flow-matching policy that predicts actions from the current observation with no future prediction, at the same size and compute. It shows how much any imagined future adds. Behavior cloning is a second floor, where published numbers exist: 16% on LIBERO-Long per ATM.

The strongest existing method is PAD, the joint diffusion model of image latents and actions. It directly embodies the claim that dense pixel prediction is necessary, and it holds the best MetaWorld number in the file at 72.5% average success over 50 tasks. We will reimplement it, check our reimplementation against that figure, and then run it at matched training compute and a matched hyperparameter-search budget. We report it at its native step count and at one and two steps. Beating it means matching its success within seed noise at native steps while using one or two network evaluations, and exceeding it when both methods are held to one or two evaluations. We found no reported PAD inference latency on Lacuna, so we will measure wall-clock time ourselves on identical hardware.

ATM is the strongest pixel-free competitor. It reaches 39% on LIBERO-Long. We will run it on LIBERO-Long with its published recipe, and include a track-target variant inside our own test bed so the architecture is held fixed. For cross-domain transfer the reference is SuSIE's 2.69 consecutive CALVIN instructions with human-video co-training. For straightening, the reference is WarmPrior applied to actions alone. We found no per-task success numbers for WarmPrior on Lacuna, so it will be rerun on Robomimic under our settings.

Two ablations isolate the key ingredient. One trains on a random subspace of the same dimension as the causal one. The other trains on the subspace that linear probes find most decodable. If either matches the causally selected target, the intervention step is not doing the work.

Why not just…?

Why not just distill PAD into one step with reflow, InstaFlow-style? This is a real competitor, and it is included as a baseline. It keeps generating full appearance latents, so every step still pays for texture. It also inherits the multi-object degradation InstaFlow reports and the collapse risk of self-generated pairs. The more important point is that nothing in the literature says a one-step distilled joint model still conditions its actions on its imagined frames. Our stage-two audit at one evaluation answers that question. If distilled PAD keeps strong motion-driven interchange effects and matches our success rate, then the selective target is unnecessary, and we will say so.

Why not just use V-JEPA features as the prediction target? Feature prediction removes pixel bias. Its own authors' critics note, though, that abstract targets lose the geometric detail needed for control, and V-JEPA has never been used as a planning target for actions. It is one of the three target variants in stage one. If its interchange effects already concentrate on motion and it performs as well as the causally selected projection, then the contribution narrows to the audit and the straightening results. That is still a useful outcome.

Why not just predict point tracks, as ATM does? Tracks are motion-only and fast, and they may turn out to be most of the answer. They are, however, a hand-chosen, 2D, sparse target, and ATM never tested which tracked points the policy relies on. Our audit will show whether the causal modes in the dense models line up with contact-region tracks. The matched-architecture track variant then shows whether anything beyond tracks, such as depth or occlusion, is being used. If tracks match our method everywhere, the conclusion is that ATM's design choice was correct for a causal reason, and our straightened track policy becomes the practical output.

Why not just drop the visual prediction at inference? Because this is the cheapest version of our hypothesis, we run it first. If PAD's actions barely change when the predicted tokens are removed at test time, the benefit is purely a training regularizer. Then the right system trains with a dense auxiliary head and generates no future at inference, and the causal-subspace policy should be judged against that simpler design.

Proposed Experiments and Decisive Checks

The primary benchmark is MetaWorld, chosen for the appearance-controlled swaps. LIBERO-Long covers long-horizon evaluation, CALVIN with Something-Something-v2 co-training covers cross-domain transfer, and Robomimic covers comparison to WarmPrior. Metrics are average task success, consecutive completed instructions on CALVIN, network evaluations and wall-clock time per action chunk, the two interchange effects, and the overlap between causal modes and probe-decodable directions. All results use at least three seeds.

Four outcomes decide the proposal. The hypothesis fails if appearance swaps move actions about as much as motion swaps in the PAD-style model. In that case dense prediction is used and the selective target should lose. The straightening question gets a clear answer if the motion interchange effect shrinks sharply at one evaluation relative to ten while success holds. That would show one-step policies stop consulting their imagination. It would be a finding about shortcut computation in straightened flows, and it would argue for history-anchored starting points as a fix, which we test directly. The method claim holds only if the causal-subspace policy at one or two evaluations matches PAD's native-step MetaWorld success and beats it at equal evaluations, while the random and probe-selected subspaces do worse. The transfer claim holds only if restricting human video to the causal subspace keeps or improves the CALVIN gain over robot-only training and closes some of the distance to SuSIE's 2.69. Losing the gain would mean appearance channels carry transferable information after all.