Making Future-Predictive Representations Actionable

Fangchuan Wang, Hao Zhang, Qiyan Zhao, Huanqi Zhao, Guanting Ye, Xu-Yao Zhang, Yuqing GaoarXiv 2026 · arXiv

TL;DR

Researchers have identified a "representation-utilization gap" in robot world models: while auxiliary future-prediction tasks help models learn rich internal representations, the action-generating parts of the network often ignore this information in deeper layers. To bridge this gap, the authors introduce RelayWAM, a model that uses lightweight "transition layers" to explicitly relay predictive cues into the action pathway, resulting in a massive boost to generalization in both simulated and real-world robot tasks without the need to generate actual future frames at inference time.

What Problem Does This Paper Address?

World Action Models (WAMs) are the current frontier in robot learning. They typically combine a large-scale video model (a "world model") with an action-generation head. The prevailing wisdom is that by training the model to predict future video frames as an auxiliary task, the network will learn a better internal understanding of physics and object dynamics, which should, in turn, make the robot's actions more robust.

However, the authors discovered a fundamental flaw in this assumption. Through a detailed layer-by-layer analysis, they found that as you go deeper into the neural network, the "future prediction" objective and the "action generation" objective start to pull the representation in different directions. While the deeper layers do contain rich information about the future, the action pathway simply doesn't "read" it. The action pathway tends to focus locally on the gripper and the immediate object, while future prediction attends to the entire scene. This "representation-utilization gap" means that much of the benefit of expensive future-prediction training is being left on the table.

The representation-utilization gap: Future information is written into deep layers, but the action pathway fails to utilize it due to a focus mismatch.

Methods

The authors first diagnose the problem using three analytical tools:

  1. Linear Probing: To see if future information is actually present in the features.
  2. Training-Focus Attribution: To map which parts of the image each objective (action vs. future) actually cares about.
  3. Causal Erasure: To test if removing future-predictive information from the features actually changes the robot's predicted actions.

To solve the gap, they propose RelayWAM. Instead of just hoping the action pathway picks up on predictive cues, they add an explicit Transition Space. This consists of recurrent transition layers that:

  • Extract: Pull step-relevant predictive cues from the intermediate and deep layers of the Video backbone.
  • Condition: Use the task instruction, robot state, and current action estimate to filter for useful future info.
  • Relay: Inject these cues directly into the attention mechanism of the Action backbone.

Crucially, this architecture follows the Future-Free Inference (FFI) paradigm. While the model is trained to predict future frames, it doesn't actually have to render those frames during deployment, keeping the system fast and efficient.

Architecture of RelayWAM, showing the transition layers bridging the Video and Action backbones.

Datasets

The model was evaluated across several rigorous benchmarks:

  • LIBERO: A standard suite for robot manipulation (Spatial, Object, Goal, and Long tasks).
  • LIBERO-Plus: Evaluates zero-shot generalization under distribution shifts (changes in background, lighting, camera view, etc.).
  • LIBERO-PRO: A more challenging generalization benchmark covering environment, language, and task variations.
  • Real-World Apple Picking: A custom setup using a Piper robotic arm to test robustness against leaf occlusion and colored ambient lighting.

Key Insights and Ideas

  • The Mismatch is Real: In standard models like Fast-WAM, the gradients for action and future prediction become nearly orthogonal in deep layers. They are essentially speaking different languages.
  • Future Info is Global, Action is Local: Future prediction needs to understand the whole scene (background, lighting) to predict pixels, but action generation is hyper-focused on the gripper-object interaction.
  • Explicit Routing is Better than Implicit Sharing: Simply sharing a backbone isn't enough. By adding a dedicated "relay" interface (which adds less than 0.6% parameter overhead), the model can finally make use of its "world knowledge."
  • Synergy: The transition space doesn't just use existing info; it actually makes the backbone more responsive to future supervision during training.
Ablation study showing that the combination of future supervision and the transition space triples the performance gain compared to supervision alone.

Key Results

RelayWAM demonstrates a significant leap in performance, particularly when things get messy (OOD scenarios):

  • LIBERO-Plus: RelayWAM achieved a 72.03% mean success rate, a massive jump from the 51.50% achieved by the previous state-of-the-art Fast-WAM.
  • Generalization: On LIBERO-Plus, adding the transition space tripled the performance gain from future supervision (from +5.2 points to +15.6 points).
  • Real-World Robustness: In apple-picking tasks under severe illumination shifts, Fast-WAM's success rate plummeted to 15%, while RelayWAM maintained a 70% success rate, rivaling models that had undergone much more extensive embodied pre-training.
Real-world apple picking results showing RelayWAM's superior robustness to lighting and placement changes.

Limitations and Future Work

While RelayWAM is highly effective, the authors note some areas for further exploration:

  • Parameter Efficiency: While the 35M parameter addition is small relative to the 6B backbone, the recurrent nature of the transition layers across depth adds some computational cost during the denoising process.
  • Long-Horizon Tasks: The current analysis focuses on relatively short-term future prediction. How this "relay" mechanism scales to very long-horizon planning remains an open question.
  • Backbone Dependency: The method relies on the quality of the underlying pretrained video model. As video models evolve (e.g., from DiT to newer architectures), the transition interface may need to be adapted.

Summary updated

Abstract

Recent world action models (WAMs) drop future prediction at inference, assuming that representations learned through auxiliary future supervision still benefit action generation. However, our layerwise analysis of Fast-WAM reveals a depth-dependent mismatch: while probes decode future state transitions more accurately from deeper visual layers, the gradients and visual focuses of the future and action objectives increasingly diverge with depth, leaving much of this predictive information unused by the action pathway. To address this issue, we introduce RelayWAM, which adds an explicit transition space between the video and action backbones. Conditioned on the task instruction, robot state, and current action estimate, recurrent transition layers extract step-relevant predictive cues across depth and relay them directly into the action attention, without generating future frames at inference. Experiments show that RelayWAM substantially amplifies the benefit of future supervision, translating into stronger generalization under distribution shifts in both simulation and real-world robot tasks. On LIBERO-Plus, RelayWAM achieves 72.03% mean success, compared to 56.42% without future supervision and 51.50% for Fast-WAM.
BibTeX
@misc{wang2026making,
  author = {Fangchuan Wang and Hao Zhang and Qiyan Zhao and Huanqi Zhao and Guanting Ye and Xu-Yao Zhang and Yuqing Gao},
  title = {{Making Future-Predictive Representations Actionable}},
  year = {2026},
  eprint = {2611.22994},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22994},
}
Figure