PhyDEX: A Benchmark for Physical Reasoning in Vision-Language-Action Models via Dynamic Extrinsic Dexterity

Hao Vo, Taisei Hanyu, Yuki Ikebe, Pham Tri Quang, Sreevenkata Anjani Tishita Godavarthi, Nghi D. Q. Bui, Daniel Sonntag, Chase Rainwater, Duy Minh Ho Nguyen, Anh Nguyen, Ngan LearXiv 2026 · arXiv

TL;DR

PhyDEX is a new benchmark designed to test whether Vision-Language-Action (VLA) models actually understand physics or are just "memorizing" visual patterns. By tasking robots with throwing objects of varying mass and friction to targets beyond their reach, the researchers found that current state-of-the-art models fail nearly 97% of the time, highlighting a critical "physical blindness" in modern AI.

What Problem Does This Paper Address?

While robotic foundation models have become impressively good at following language instructions like "pick up the apple," they are largely evaluated on simple "pick-and-place" tasks. These tasks focus on the contact between the gripper and the object but ignore the underlying Newtonian mechanics—gravity, inertia, friction, and mass distribution—that govern real-world interaction.

The authors argue that current models lack dynamic extrinsic dexterity: the ability to use external physical forces to manipulate objects. If a robot doesn't know that a box with a lead weight in one corner will tumble differently than an empty one, it can't perform complex, high-speed tasks like throwing. PhyDEX aims to bridge this gap by forcing models to infer hidden physical properties to succeed.

Methods

The researchers developed a fully automated pipeline to generate a massive dataset of successful throwing trajectories. The process involves:

  1. Grasp Generation: Using GraspGen-X to find geometrically diverse and stable ways to hold an object.
  2. Motion Planning: Utilizing cuRobo for collision-free movement.
  3. Throw Optimization: A closed-loop optimizer (SLSQP) calculates the exact joint velocities and release timing needed to hit a target, accounting for the object's specific mass, center of mass (CoM), and friction.

To test for true reasoning, they applied heavy domain randomization. They varied object mass (0.5x to 2.0x), offset the CoM by up to 15mm, and scaled friction coefficients. This ensures that two objects might look identical but require completely different throwing strategies.

The PhyDEX data generation pipeline, from environment construction to trajectory optimization.

Datasets

The paper introduces the PhyDEX benchmark, which consists of:

  • 35,357 trajectories (approx. 143 hours of data).
  • 9 distinct tasks across 100 kitchen scenes.
  • Three object categories: Uniform, Irregular (offset CoM), and Dynamic (mass changes during the task, like a bottle being emptied).
  • Three destination types: Static, Multiple, and Moving targets.
PhyDEX task taxonomy showing the distribution of object and destination types.

Key Insights and Ideas

  • Physical Blindness: Current VLA models are "physically blind." Because mass and CoM aren't directly visible in an image, models treat visually similar objects the same way, leading to consistent failure when physics vary.
  • Grasping is a Bottleneck: Surprisingly, 89% of failures occur before the throw even happens. Current pretrained models aren't robust enough to handle the visual and positional diversity of the benchmark, failing at the reaching or grasping stage.
  • The Power of Priors: When the researchers "cheated" and gave the models ground-truth physical data (like the exact mass), success rates nearly doubled. This suggests that the models can learn the dynamics; they just can't infer the properties from vision alone.
  • Force Sensing Helps, but isn't a Silver Bullet: Models with force/torque sensors (like ForceVLA) performed better in the real world because they can "feel" the weight once grasped, but they still struggle with complex distributions like offset CoMs.

Key Results

The evaluation of three representative models—GR00T N1.7, DynamicVLA, and ForceVLA—yielded sobering results:

  • Overall Success Rate: All models scored below 3% on the full benchmark.
  • Complexity Penalty: Success rates dropped to near zero for irregular and dynamic-mass objects.
  • Real-World Validation: In a physical setup using 3D-printed cubes with hidden weights, ForceVLA achieved a 51.3% success rate, but its performance plummeted as soon as the object's physics deviated from the "standard" training cube.
Comparison of success rates with and without ground-truth physics injection.

Limitations and Future Work

The authors acknowledge several limitations:

  • Skill Scope: The benchmark currently only focuses on throwing. Future versions should include pushing, sliding, and pouring.
  • Simplified Physics: The simulation does not yet account for aerodynamics (important for light objects) or elasticity (bouncing).
  • Model Variety: Only three VLA architectures were tested; a broader range of models might reveal different scaling properties.

The team suggests that future robot "brains" need explicit mechanisms for probing—interacting with an object specifically to learn its weight and friction before attempting a complex maneuver.

Summary updated

Abstract

Vision-Language-Action (VLA) models have advanced rapidly in language-conditioned robotic manipulation, yet existing benchmarks mainly evaluate gripper--object contact and largely ignore how objects interact with the environment through gravity, friction, inertia, and mass distribution. As a result, it remains unclear whether current policies truly understand physics or merely interpolate from their training data. We address this gap through the lens of dynamic extrinsic dexterity and introduce PhyDEX, a throwing benchmark in which a robot must infer an object's mass, center of mass, and friction to send it to a destination beyond its kinematic workspace. PhyDEX contains 35K trajectories (143 hours) across nine tasks and 100 scenes, covering uniform, irregular, and dynamic-mass objects with static, multiple, and moving destinations. All data are generated by a scalable, fully automatic pipeline that combines grasp generation, motion planning, and closed-loop throw optimization under randomized physical properties. We evaluate three representative VLAs, GR00T N1.7, DynamicVLA, and ForceVLA, and find that all achieve success rates below 3\%. Our analysis reveals two key bottlenecks: limited robustness of pretrained models, which causes most failures before the throw, and physical blindness, as providing ground-truth physical properties roughly doubles success. Experiments on a humanoid robot and in the real world show the same failure patterns. We hope PhyDEX will drive the development of physically grounded robot policies.
BibTeX
@misc{vo2026phydex,
  author = {Hao Vo and Taisei Hanyu and Yuki Ikebe and Pham Tri Quang and Sreevenkata Anjani Tishita Godavarthi and Nghi D. Q. Bui and Daniel Sonntag and Chase Rainwater and Duy Minh Ho Nguyen and Anh Nguyen and Ngan Le},
  title = {{PhyDEX: A Benchmark for Physical Reasoning in Vision-Language-Action Models via Dynamic Extrinsic Dexterity}},
  year = {2026},
  eprint = {2611.26565},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26565},
}
Figure