TL;DR
World Action Models (WAMs) gain their impressive generalization from predicting the future, but the iterative video denoising required at inference makes them painfully slow. This paper reveals a surprising "aha" moment: you don't actually need to generate clean future frames to get the benefits—a single forward pass over fully noised future tokens is sufficient. The authors introduce Simple-WAM, which matches the speed of fast "latent" models while significantly outperforming them in robustness and task generalization.
What Problem Does This Paper Address?
In embodied AI, we want robot policies that don't just memorize training data but generalize to new environments and tasks. World Action Models (WAMs) attempt this by jointly predicting future video frames and robot actions. By leveraging large-scale video pre-training, WAMs inherit rich spatiotemporal priors about how the world moves.
However, there is a major efficiency bottleneck. Explicit WAMs perform iterative denoising at test-time to generate clean future frames, which is computationally expensive and slow. To fix this, researchers proposed Latent WAMs, which predict the future during training but discard the future-modeling branch entirely at inference for speed. While Latent WAMs perform well on tasks they've seen before, this paper asks a critical question: Does discarding the future at test-time kill the very generalization that makes WAMs special?
Methods
The authors conducted a controlled empirical study comparing explicit and latent paradigms across three axes: environmental perturbation, data efficiency, and task generalization.
Their investigation led to two key findings:
- Generalization Gap: Latent WAMs match explicit ones in-distribution but fail significantly when the environment changes or data is scarce.
- The "Noisy" Shortcut: Most of the generalization benefit comes from the first denoising step. The action expert needs the representation of the future, not the final clean pixels.
Based on this, they proposed Simple-WAM:
- Inference: Instead of 10+ denoising steps, it performs a single forward pass where future video tokens are kept as pure Gaussian noise ($\tau=1$). The action expert conditions on these noisy representations.
- Training: They introduce a Mixed Schedule. During training, the model is occasionally forced to predict from pure noise (probability $p=0.5$). This "sharpens" the video expert's ability to extract useful features from noise, aligning training behavior with the simplified inference.
Datasets
The study utilizes several rigorous benchmarks:
- LIBERO: A suite for spatial, object-centric, and long-horizon robot tasks.
- LIBERO-Plus: A benchmark specifically designed to test robustness against seven types of environmental perturbations (e.g., lighting, background, camera angles).
- RoboTwin 2.0: A bimanual manipulation benchmark.
- Real-World Evaluation: Experiments conducted on an AgileX Aloha dual-arm platform across four tasks, including a held-out "Store in Order" task.
Key Insights and Ideas
- Preparing vs. Generating: The paper argues that the benefit of WAMs comes from preparing the future. The video backbone provides a "template" of temporal evolution that the action expert uses to ground its motor commands.
- In-Distribution is Deceptive: If you only look at success rates on training tasks, Latent WAMs look great. The "Generalization Deteriorates" finding (F1 in Figure 1) shows that future modeling is a test-time requirement for robustness, not just a training objective.
- Noise as a Feature: Surprisingly, pure Gaussian noise tokens, when processed by a pre-trained video DiT (Diffusion Transformer), provide enough spatiotemporal context for the action expert to generalize. Replacing these with zeros or learnable embeddings causes performance to crater, proving that the video backbone's pre-trained knowledge is specifically tied to handling noised inputs.
Key Results
- Speed: Simple-WAM runs at 74.7 ms per action chunk, which is 3.8x faster than explicit WAMs (286.9 ms) and nearly as fast as latent models (62.0 ms).
- Robustness: On LIBERO-Plus (environmental perturbations), Simple-WAM achieved a 79.5% success rate, crushing the latent paradigm (53.8%) and even beating the explicit paradigm (67.7%).
- Task Generalization: When given action-free video of a new task, Simple-WAM's success rate jumped from ~2% to 73.6%, whereas the latent model remained stuck at ~6%.
- Real-World: In physical robot tests, Simple-WAM maintained a 69.2% success rate under perturbations, while the latent model dropped to 25.2%.
Limitations and Future Work
- Backbone Specificity: The study primarily uses a 5B-parameter Wan2.2 video DiT. While the findings are robust, it's unclear if they hold for significantly smaller or differently architected backbones.
- Scaling: The authors note that whether these conclusions hold at even larger scales (e.g., 10B+ parameters) or with massive embodied pre-training remains an open question.
- Future Direction: Exploring how to further "distill" the spatiotemporal features from the video expert to reduce the 1.2x latency overhead compared to latent models.
Summary updated