TL;DR
Verify2Act is a modular framework that gives Vision-Language Models (VLMs) a "physical sanity check" by using a latent world model to imagine the consequences of proposed actions. By verifying plans in a compressed feature space rather than generating raw pixels, it achieves 83.3% success on real-world robot tasks with zero planning-induced failures and significantly higher efficiency than diffusion-based alternatives.
What Problem Does This Paper Address?
Large Vision-Language Models (VLMs) like GPT-4o are excellent at high-level semantic reasoning—they can easily tell a robot to "pick up the red block and put it in the bin." However, they lack a grounded understanding of physics. They don't know if a block is reachable, if it's occluded, or if a specific movement will result in a collision.
To fix this, researchers often use "world models" to predict future video frames. But pixel-level prediction is computationally expensive, slow, and prone to "visual drift," where the generated images become blurry or nonsensical over long sequences. Verify2Act addresses this by moving the verification process into a latent feature space, allowing the robot to "imagine" the results of its actions quickly and accurately without ever needing to render a single pixel.
Methods
The Verify2Act framework operates in a three-stage loop:
- Semantic Action Proposal: A VLM (like GPT-4o) acts as a high-level heuristic, proposing a small set of candidate multi-step plans based on the current camera view and a language goal.
- Latent Imagination (V2A-WM): This is the core of the paper. The system uses a world model called V2A-WM to "roll out" these plans in the latent feature space of DINOv2. Unlike previous models, V2A-WM uses cross-attention action grounding to link specific language tokens (from CLIP) to visual patches, and a causal temporal history window to remember objects even when they are hidden by the robot's gripper.
- Dual-Head Critic Evaluation: A specialized critic scores the imagined rollouts. One head checks for temporal consistency (does the sequence look physically plausible?), while the other checks for goal satisfaction (does the final state actually match the user's request?).
The entire system is trained in a "digital twin" (a simulation of the real workspace) and transferred to the real robot using an optional real2sim alignment step to ensure the features match.
Datasets
The authors evaluated Verify2Act across three distinct environments:
- Real-World Robot: A low-cost Yahboom DOFBOT arm performing six language-conditioned tasks (e.g., bin clearing, stacking, spatial relations).
- CALVIN: A standard benchmark for long-horizon, language-conditioned manipulation.
- Robosuite (Cluttered Nut Assembly): A challenging simulation environment requiring precise manipulation in crowded scenes.
Key Insights and Ideas
- Latent vs. Pixel Space: Verifying in the DINOv2 latent space is roughly 60x faster than pixel-diffusion models (0.08s vs 5.1s per rollout). It also avoids the "hallucinations" common in generative video models.
- The Power of the Critic: The VLM is already a decent planner (93.3% plan correctness on simple tasks). The real value of Verify2Act isn't just "fixing" bad plans, but reliably filtering out the small fraction of incorrect ones before the robot moves.
- Handling Occlusion: By using a temporal history window, the model avoids "amnesia." When the gripper closes around a block, the model remembers the block is still there because it "sees" the history of the scene.
- Sim-to-Real via Digital Twins: The world model and critic were trained entirely on synthetic data from a digital twin, yet they generalized to real-world camera frames with high accuracy.
Key Results
- Real-World Success: Verify2Act achieved an 83.3% task success rate, significantly outperforming the VLM-only baseline (53.3%) and a pixel-diffusion world model (46.7%).
- Zero Planning Failures: On the robot, 100% of the plans executed by Verify2Act were semantically and physically correct. Every single failure was due to low-level execution slips (e.g., a block dropping), not a bad plan.
- Efficiency: In the CALVIN simulation, Verify2Act reached a 75% success rate on 5-task chains while requiring 30x fewer calls to the expensive VLM compared to reflective diffusion-based methods.
- Robustness: Even without the real2sim alignment step, the model ranked the correct plan first in 71% of real-world scenes, proving the latent features are robust to visual domain shifts.
Limitations and Future Work
- Task Complexity: The current evaluation is limited to tabletop block manipulation. Future work is needed to test broader geometries and contact-rich tasks (like opening drawers or using tools).
- Alignment Overhead: The real2sim alignment step, while helpful, adds several seconds of processing time per frame.
- Dynamic Backgrounds: The "background-sparsity mask" used to keep the world model stable assumes a static camera and background. This might fail if the camera moves or if there are other moving objects in the scene.
- Low-Level Skills: Since the framework is "policy-agnostic," it still relies on the quality of the underlying low-level controllers. Improving these controllers would likely push the 83.3% success rate even higher.
Summary updated