DiffWAM: Learning World Action Model from Consecutive Visual Difference

Xinheng Zhang, Haoyu Xiong, Chao Jing, Xiaohan Zhang, Linshan Li, Hongbang Fan, Xingyu ChenarXiv 2026 · arXiv

TL;DR

DiffWAM is a new "World Action Model" that improves robot learning by forcing the model to explicitly predict the visual changes between consecutive video frames. By focusing on these "visual differences" during training, the model learns more task-relevant representations, leading to state-of-the-art performance on benchmarks and a massive 15% success rate boost in real-world manipulation tasks—all with zero extra computational cost at runtime.

What Problem Does This Paper Address?

In the quest to build general-purpose robots, World Action Models (WAMs) have become a popular strategy. These models learn by trying to predict two things simultaneously: what the robot should do next (actions) and what the world will look like after that action (future frames).

The problem is that predicting a full future frame is incredibly "noisy." If a robot is picking up a pen in a cluttered room, 95% of the pixels in the video—the table, the walls, the lighting—stay exactly the same. Standard WAMs struggle to distinguish between these static, task-irrelevant details and the subtle, crucial changes caused by the robot's movement. Because the model is rewarded for predicting the whole frame, it often gets "distracted" by the background and fails to capture the fine-grained dynamics of the actual task.

Methods

The authors introduce DiffWAM, which uses a Mixture-of-Experts (MoE) framework to sharpen the model's focus. The architecture consists of a shared Observation Encoder and three specialized experts:

  1. World Expert: The standard component that predicts the full future state (using denoising or flow matching).
  2. Action Expert: The component that generates the robot's trajectory.
  3. Diff Expert (The Secret Sauce): This new expert is tasked with predicting the latent visual difference ($\Delta f$) between consecutive frames. Instead of imagining the whole scene, it must specifically predict how the latent representation of the world changes from time $t$ to $t+1$.

To keep these experts from interfering with each other while still benefiting from a shared understanding of the scene, the authors use a structured unidirectional attention mask. This ensures that while all experts use the features from the shared encoder, the "Diff" and "World" predictions don't "leak" into the action generation pathway in a way that would require them to be present during deployment.

The "Free Lunch": At inference time (when the robot is actually working), the World and Diff experts are discarded. Only the Observation Encoder and Action Expert remain. This means the robot gets the benefit of a "smarter" internal representation learned during training without any extra latency or memory overhead during execution.

Datasets

The researchers evaluated DiffWAM across three major settings:

  • LIBERO: A benchmark featuring 40 single-arm manipulation tasks (e.g., opening drawers, picking up objects).
  • RoboTwin 2.0: A challenging suite of 50 bimanual (two-armed) tasks, tested in both "clean" environments and "randomized" scenes to check for out-of-distribution robustness.
  • Real-World Evaluation: Conducted on the ALOHA-AgileX platform, focusing on three tasks: Place Pens, Store Block, and Stack Cups.

Key Insights and Ideas

  • Visual Foresight is about Change: The core "aha moment" is that a robot doesn't just need to know what the future looks like; it needs to know how the world is changing. Explicitly supervising the "diff" forces the encoder to prioritize the moving parts of the scene.
  • Attention Matters: Qualitative visualizations show that DiffWAM’s attention is much more "focused" on the objects being manipulated compared to baselines like FastWAM, which often show "diffused" or "distracted" attention patterns.
  • Regularization via Freezing: For small datasets, the authors found that freezing the final layers of the Diff Expert (DiffWAM-S) acted as a powerful regularizer, preventing the model from overfitting to visual noise while still allowing the "diff" signal to shape the shared encoder.

Key Results

DiffWAM consistently outperformed the state-of-the-art baseline, FastWAM:

  • Benchmark Superiority: Achieved an average success rate of 98.8% on LIBERO and 92.65% on RoboTwin 2.0.
  • Real-World Leap: In physical robot tests, DiffWAM reached an 88.3% success rate, a significant 15.0 percentage point improvement over FastWAM (73.3%).
  • Training Efficiency: DiffWAM reached a 95% success rate on LIBERO 28.1% faster than the baseline (6.9k steps vs 9.6k steps), proving that the "diff" signal is a much more efficient way to learn.

Limitations and Future Work

While impressive, the authors note a few hurdles:

  • Training Cost: Adding the Diff Expert increases the VRAM and computational requirements during the training phase.
  • Signal Ambiguity: The current "diff" target doesn't distinguish between changes caused by the robot and "nuisance" changes like camera shake or shifting shadows.
  • Scaling: The model hasn't yet been tested on massive, "internet-scale" robot datasets.

Future work will look into multi-scale and multi-horizon diff prediction (predicting changes over longer time jumps) and incorporating object-centric constraints to better disentangle the robot's actions from the rest of the environment.

Summary updated

Abstract

World Action Models (WAMs) have emerged as an important paradigm in robot learning, jointly predicting future scene states and robot actions from current observations and instructions. However, future states entangle task-related changes with large amounts of task-irrelevant content, such as static backgrounds, which weakens the model's ability to capture changes relevant to task execution. To address this, we propose Difference World Action Model (DiffWAM), a Mixture-of-Experts (MoE) framework that enhances world representations by jointly learning future-state prediction and consecutive visual difference prediction. We define consecutive visual differences as differences between the latent representations of consecutive observations, providing an explicit learning signal for temporal variation. The framework comprises a World Expert for future frame prediction, an Action Expert for action generation, and a Diff Expert for predicting these latent differences along the trajectory. A structured unidirectional attention mask conditions the Diff, World, and Action Experts solely on the current-frame representation while blocking attention between experts. This design allows both the World and Diff Experts to be removed during inference, introducing no additional inference overhead. DiffWAM achieves state-of-the-art performance on the LIBERO and RoboTwin 2.0 benchmarks and substantially improves over FastWAM on real-world manipulation tasks. Specifically, it attains average success rates of 98.8%, 92.65%, and 88.3%, outperforming the state-of-the-art baseline (i.e., FastWAM) by 1.75, 1.55, and 15.0 percentage points, respectively.
BibTeX
@misc{zhang2026diffwam,
  author = {Xinheng Zhang and Haoyu Xiong and Chao Jing and Xiaohan Zhang and Linshan Li and Hongbang Fan and Xingyu Chen},
  title = {{DiffWAM: Learning World Action Model from Consecutive Visual Difference}},
  year = {2026},
  eprint = {2611.19904},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19904},
}
Figure