PhysVAM: Physics-oriented Future Value Action Model for Dynamic Object Manipulation

Yuxiang Yang, Jingbo Zhang, Jiaru Zhong, Junliang Chen, Yirui Zhou, Guangming Wang, Haitao Lin, Huaiyuan Xu, Yi Wang, Lap-Pui ChauarXiv 2026 · arXiv

In the rapidly evolving field of robotic manipulation, the shift from static tabletops to dynamic environments—where objects move, roll, or rotate during the robot's very attempt to grab them—presents a formidable challenge. Current "World Action Models" often try to solve this by "hallucinating" future video frames to anticipate motion. However, as any researcher in the field knows, pixels are messy, and geometric distortions in generated video can lead to catastrophic failures in physical interaction.

PhysVAM (Physics-oriented Future Value Action Model) offers a clever pivot: instead of trying to predict what the future will look like, why not predict how well the task is going?

TL;DR

PhysVAM introduces a "task-value" perspective for dynamic robotic manipulation, replacing computationally expensive visual future reconstruction with multi-horizon future value prediction. By integrating these anticipated task-progress representations into a Diffusion Transformer policy, the model achieves a staggering ~27% absolute improvement in success rates over state-of-the-art baselines in both simulation and real-world dynamic tasks.

What Problem Does This Paper Address?

Dynamic manipulation is a race against time and physics. If a robot perceives an object at time $t$, by the time it executes an action at $t+1$, a rolling ball or a rotating turntable has already moved the target.

Existing approaches generally fall into two camps:

  1. Visual Future Modeling: These models predict future video frames. While intuitive, they often suffer from "hallucinations"—objects might jump positions or distort shapes in the predicted frames, making them unreliable for precise contact.
  2. Current-Value Assessment: These models evaluate how "good" the current state is. However, a high current value doesn't guarantee success; a robot might be perfectly aligned now, but if it doesn't anticipate the object moving away before the gripper closes, it will still miss.

PhysVAM addresses this by learning to anticipate how future interactions will advance or hinder task progress across multiple time horizons, providing "task-directed foresight" without the overhead of pixel-perfect reconstruction.

Comparison of visual vs. value-based future modeling. Visual prediction (a) can hallucinate, while PhysVAM (b) predicts task progress to guide actions.

Methods

The architecture of PhysVAM is built on two coupled branches:

  1. ValueDiT (The Critic): A Diffusion Transformer that learns to predict task values at multiple future horizons ($h = 4, 8, 16, 32$ steps). It is trained on both successful and failed trajectories, allowing it to recognize the "latent signatures" of an impending failure or a successful grasp.
  2. ActionDiT (The Actor): A 1B-parameter Diffusion Transformer (initialized from pretrained video DiT weights) that generates 32-step action chunks.

The Secret Sauce: Gated Cross-Attention The latent representations from the ValueDiT are injected into the ActionDiT layers through a gated cross-attention mechanism. This allows the action policy to "listen" to the value branch’s foresight. If the value branch senses a "dip" in future progress (e.g., a missed grasp), the action branch can adjust its trajectory in real-time to compensate.

Historical Motion Encoding Unlike models that rely solely on current frames, PhysVAM explicitly encodes the last four frames of 3D object trajectories and robot actions. This provides the necessary "physics context"—velocity and acceleration—to handle high-speed dynamic targets.

The PhysVAM framework, showing the dual-branch architecture and the integration of historical motion and multi-horizon value queries.

Datasets

  • Simulation: The authors reconstructed dynamic scenarios using ten geometrically diverse objects (spheres, bottles, cups, etc.). The dataset includes 12K successful and 10K failed trajectories, specifically designed to emphasize high-speed interactions ($v \ge 0.4$ m/s).
  • Real-World: Experiments were conducted on an AgileX PiPER arm across three challenging scenarios: conveyor-belt translation, turntable rotation, and object rolling.
  • Object Tracking: In the real world, the system uses FoundationPose to estimate 6D poses from RGB-D streams, which are then converted into the historical motion tokens required by the model.

Key Insights and Ideas

  • Value as a Compact Representation: Predicting a scalar value (or its latent representation) is far more efficient and robust for control than predicting high-dimensional video. It filters out irrelevant visual noise and focuses strictly on task-relevant physics.
  • Multi-Horizon Supervision: By supervising at multiple offsets ($h=4$ to $h=32$), the model learns both immediate reactive adjustments and longer-term strategic alignment.
  • Learning from Failure: A key "aha" moment is that while the action branch only learns from successful demonstrations, the value branch learns from both success and failure. This allows the model to recognize "danger zones" in the state space that a success-only model would never see.
  • Transferability: The authors demonstrated that their value-guidance branch could be "plugged into" other models like PUMA and DynamicVLA, significantly boosting their performance as well.

Key Results

PhysVAM sets a new high bar for dynamic manipulation:

  • Simulation: Achieved a 58.83% success rate, outperforming the strongest baseline (PUMA at 31.83%) by 27.0 percentage points.
  • Real-World: Achieved a 64.17% success rate, outperforming PUMA (36.67%) by 27.5 percentage points.
  • Efficiency: It recorded the shortest average completion time (8.23s), indicating more decisive and coordinated movements.
Performance comparison showing PhysVAM's superior success rates in both simulation and real-world tasks.

Limitations and Future Work

While impressive, the authors note a few areas for growth:

  • Offline Supervision: The current framework relies on pre-labeled offline trajectories. Future iterations could move toward online self-improvement, where the robot uses its own value predictions to guide exploration and reinforcement learning.
  • Complexity: The current tasks involve single-object manipulation. Moving toward multi-object interactions and longer-horizon tasks (e.g., sequential assembly of moving parts) remains an open challenge.
  • Hybrid Models: There is potential in combining PhysVAM’s value guidance with visual world models to get the best of both worlds: explicit scene understanding and task-directed foresight.

Summary updated

Abstract

Predicting future scene or object states does not necessarily reveal how robot-object interactions will advance or hinder a task. This challenge is particularly pronounced in dynamic robotic manipulation, where objects continue moving during perception and execution. Effective action generation therefore requires not only modeling object dynamics but also anticipating how future interactions will shape task progress. To this end, we propose \textbf{PhysVAM}, a Physics-oriented Future Value Action Model that represents anticipated interactions through their effects on future task progress. This value-based formulation provides task-directed foresight without reconstructing future visual states. Conditioned on current observations and historical object and robot motion, PhysVAM jointly learns action generation and multi-horizon future-value prediction. The value branch learns horizon-specific latent representations through future task-value prediction from successful and failed trajectories. These future-value representations are integrated into the action policy through gated cross-attention, guiding action generation toward interactions that facilitate task success. Extensive simulation and real-world experiments demonstrate superior task success, with absolute success-rate improvements of 27.0 and 27.5 percentage points over the strongest baselines in simulation and real-world tasks, respectively. Project page: https://metarobo.github.io/physvam.github.io/.
BibTeX
@misc{yang2026physvam,
  author = {Yuxiang Yang and Jingbo Zhang and Jiaru Zhong and Junliang Chen and Yirui Zhou and Guangming Wang and Haitao Lin and Huaiyuan Xu and Yi Wang and Lap-Pui Chau},
  title = {{PhysVAM: Physics-oriented Future Value Action Model for Dynamic Object Manipulation}},
  year = {2026},
  eprint = {2611.26105},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26105},
}
Figure