TL;DR
Researchers have identified a "representation-utilization gap" in robot world models: while auxiliary future-prediction tasks help models learn rich internal representations, the action-generating parts of the network often ignore this information in deeper layers. To bridge this gap, the authors introduce RelayWAM, a model that uses lightweight "transition layers" to explicitly relay predictive cues into the action pathway, resulting in a massive boost to generalization in both simulated and real-world robot tasks without the need to generate actual future frames at inference time.
What Problem Does This Paper Address?
World Action Models (WAMs) are the current frontier in robot learning. They typically combine a large-scale video model (a "world model") with an action-generation head. The prevailing wisdom is that by training the model to predict future video frames as an auxiliary task, the network will learn a better internal understanding of physics and object dynamics, which should, in turn, make the robot's actions more robust.
However, the authors discovered a fundamental flaw in this assumption. Through a detailed layer-by-layer analysis, they found that as you go deeper into the neural network, the "future prediction" objective and the "action generation" objective start to pull the representation in different directions. While the deeper layers do contain rich information about the future, the action pathway simply doesn't "read" it. The action pathway tends to focus locally on the gripper and the immediate object, while future prediction attends to the entire scene. This "representation-utilization gap" means that much of the benefit of expensive future-prediction training is being left on the table.
Methods
The authors first diagnose the problem using three analytical tools:
- Linear Probing: To see if future information is actually present in the features.
- Training-Focus Attribution: To map which parts of the image each objective (action vs. future) actually cares about.
- Causal Erasure: To test if removing future-predictive information from the features actually changes the robot's predicted actions.
To solve the gap, they propose RelayWAM. Instead of just hoping the action pathway picks up on predictive cues, they add an explicit Transition Space. This consists of recurrent transition layers that:
- Extract: Pull step-relevant predictive cues from the intermediate and deep layers of the Video backbone.
- Condition: Use the task instruction, robot state, and current action estimate to filter for useful future info.
- Relay: Inject these cues directly into the attention mechanism of the Action backbone.
Crucially, this architecture follows the Future-Free Inference (FFI) paradigm. While the model is trained to predict future frames, it doesn't actually have to render those frames during deployment, keeping the system fast and efficient.
Datasets
The model was evaluated across several rigorous benchmarks:
- LIBERO: A standard suite for robot manipulation (Spatial, Object, Goal, and Long tasks).
- LIBERO-Plus: Evaluates zero-shot generalization under distribution shifts (changes in background, lighting, camera view, etc.).
- LIBERO-PRO: A more challenging generalization benchmark covering environment, language, and task variations.
- Real-World Apple Picking: A custom setup using a Piper robotic arm to test robustness against leaf occlusion and colored ambient lighting.
Key Insights and Ideas
- The Mismatch is Real: In standard models like Fast-WAM, the gradients for action and future prediction become nearly orthogonal in deep layers. They are essentially speaking different languages.
- Future Info is Global, Action is Local: Future prediction needs to understand the whole scene (background, lighting) to predict pixels, but action generation is hyper-focused on the gripper-object interaction.
- Explicit Routing is Better than Implicit Sharing: Simply sharing a backbone isn't enough. By adding a dedicated "relay" interface (which adds less than 0.6% parameter overhead), the model can finally make use of its "world knowledge."
- Synergy: The transition space doesn't just use existing info; it actually makes the backbone more responsive to future supervision during training.
Key Results
RelayWAM demonstrates a significant leap in performance, particularly when things get messy (OOD scenarios):
- LIBERO-Plus: RelayWAM achieved a 72.03% mean success rate, a massive jump from the 51.50% achieved by the previous state-of-the-art Fast-WAM.
- Generalization: On LIBERO-Plus, adding the transition space tripled the performance gain from future supervision (from +5.2 points to +15.6 points).
- Real-World Robustness: In apple-picking tasks under severe illumination shifts, Fast-WAM's success rate plummeted to 15%, while RelayWAM maintained a 70% success rate, rivaling models that had undergone much more extensive embodied pre-training.
Limitations and Future Work
While RelayWAM is highly effective, the authors note some areas for further exploration:
- Parameter Efficiency: While the 35M parameter addition is small relative to the 6B backbone, the recurrent nature of the transition layers across depth adds some computational cost during the denoising process.
- Long-Horizon Tasks: The current analysis focuses on relatively short-term future prediction. How this "relay" mechanism scales to very long-horizon planning remains an open question.
- Backbone Dependency: The method relies on the quality of the underlying pretrained video model. As video models evolve (e.g., from DiT to newer architectures), the transition interface may need to be adapted.
Summary updated