TL;DR
PhyDEX is a new benchmark designed to test whether Vision-Language-Action (VLA) models actually understand physics or are just "memorizing" visual patterns. By tasking robots with throwing objects of varying mass and friction to targets beyond their reach, the researchers found that current state-of-the-art models fail nearly 97% of the time, highlighting a critical "physical blindness" in modern AI.
What Problem Does This Paper Address?
While robotic foundation models have become impressively good at following language instructions like "pick up the apple," they are largely evaluated on simple "pick-and-place" tasks. These tasks focus on the contact between the gripper and the object but ignore the underlying Newtonian mechanics—gravity, inertia, friction, and mass distribution—that govern real-world interaction.
The authors argue that current models lack dynamic extrinsic dexterity: the ability to use external physical forces to manipulate objects. If a robot doesn't know that a box with a lead weight in one corner will tumble differently than an empty one, it can't perform complex, high-speed tasks like throwing. PhyDEX aims to bridge this gap by forcing models to infer hidden physical properties to succeed.
Methods
The researchers developed a fully automated pipeline to generate a massive dataset of successful throwing trajectories. The process involves:
- Grasp Generation: Using GraspGen-X to find geometrically diverse and stable ways to hold an object.
- Motion Planning: Utilizing cuRobo for collision-free movement.
- Throw Optimization: A closed-loop optimizer (SLSQP) calculates the exact joint velocities and release timing needed to hit a target, accounting for the object's specific mass, center of mass (CoM), and friction.
To test for true reasoning, they applied heavy domain randomization. They varied object mass (0.5x to 2.0x), offset the CoM by up to 15mm, and scaled friction coefficients. This ensures that two objects might look identical but require completely different throwing strategies.
Datasets
The paper introduces the PhyDEX benchmark, which consists of:
- 35,357 trajectories (approx. 143 hours of data).
- 9 distinct tasks across 100 kitchen scenes.
- Three object categories: Uniform, Irregular (offset CoM), and Dynamic (mass changes during the task, like a bottle being emptied).
- Three destination types: Static, Multiple, and Moving targets.
Key Insights and Ideas
- Physical Blindness: Current VLA models are "physically blind." Because mass and CoM aren't directly visible in an image, models treat visually similar objects the same way, leading to consistent failure when physics vary.
- Grasping is a Bottleneck: Surprisingly, 89% of failures occur before the throw even happens. Current pretrained models aren't robust enough to handle the visual and positional diversity of the benchmark, failing at the reaching or grasping stage.
- The Power of Priors: When the researchers "cheated" and gave the models ground-truth physical data (like the exact mass), success rates nearly doubled. This suggests that the models can learn the dynamics; they just can't infer the properties from vision alone.
- Force Sensing Helps, but isn't a Silver Bullet: Models with force/torque sensors (like ForceVLA) performed better in the real world because they can "feel" the weight once grasped, but they still struggle with complex distributions like offset CoMs.
Key Results
The evaluation of three representative models—GR00T N1.7, DynamicVLA, and ForceVLA—yielded sobering results:
- Overall Success Rate: All models scored below 3% on the full benchmark.
- Complexity Penalty: Success rates dropped to near zero for irregular and dynamic-mass objects.
- Real-World Validation: In a physical setup using 3D-printed cubes with hidden weights, ForceVLA achieved a 51.3% success rate, but its performance plummeted as soon as the object's physics deviated from the "standard" training cube.
Limitations and Future Work
The authors acknowledge several limitations:
- Skill Scope: The benchmark currently only focuses on throwing. Future versions should include pushing, sliding, and pouring.
- Simplified Physics: The simulation does not yet account for aerodynamics (important for light objects) or elasticity (bouncing).
- Model Variety: Only three VLA architectures were tested; a broader range of models might reveal different scaling properties.
The team suggests that future robot "brains" need explicit mechanisms for probing—interacting with an object specifically to learn its weight and friction before attempting a complex maneuver.
Summary updated