Verify2Act: Critic-Guided Latent World Models for Verifying Language-Conditioned Manipulation Plans

Chrisantus Eze, Christopher CrickarXiv 2026 · arXiv

TL;DR

Verify2Act is a modular framework that gives Vision-Language Models (VLMs) a "physical sanity check" by using a latent world model to imagine the consequences of proposed actions. By verifying plans in a compressed feature space rather than generating raw pixels, it achieves 83.3% success on real-world robot tasks with zero planning-induced failures and significantly higher efficiency than diffusion-based alternatives.

What Problem Does This Paper Address?

Large Vision-Language Models (VLMs) like GPT-4o are excellent at high-level semantic reasoning—they can easily tell a robot to "pick up the red block and put it in the bin." However, they lack a grounded understanding of physics. They don't know if a block is reachable, if it's occluded, or if a specific movement will result in a collision.

To fix this, researchers often use "world models" to predict future video frames. But pixel-level prediction is computationally expensive, slow, and prone to "visual drift," where the generated images become blurry or nonsensical over long sequences. Verify2Act addresses this by moving the verification process into a latent feature space, allowing the robot to "imagine" the results of its actions quickly and accurately without ever needing to render a single pixel.

Methods

The Verify2Act framework operates in a three-stage loop:

  1. Semantic Action Proposal: A VLM (like GPT-4o) acts as a high-level heuristic, proposing a small set of candidate multi-step plans based on the current camera view and a language goal.
  2. Latent Imagination (V2A-WM): This is the core of the paper. The system uses a world model called V2A-WM to "roll out" these plans in the latent feature space of DINOv2. Unlike previous models, V2A-WM uses cross-attention action grounding to link specific language tokens (from CLIP) to visual patches, and a causal temporal history window to remember objects even when they are hidden by the robot's gripper.
  3. Dual-Head Critic Evaluation: A specialized critic scores the imagined rollouts. One head checks for temporal consistency (does the sequence look physically plausible?), while the other checks for goal satisfaction (does the final state actually match the user's request?).
Overview of the Verify2Act framework showing the three-stage pipeline: Proposal, Imagination, and Evaluation.

The entire system is trained in a "digital twin" (a simulation of the real workspace) and transferred to the real robot using an optional real2sim alignment step to ensure the features match.

Datasets

The authors evaluated Verify2Act across three distinct environments:

  • Real-World Robot: A low-cost Yahboom DOFBOT arm performing six language-conditioned tasks (e.g., bin clearing, stacking, spatial relations).
  • CALVIN: A standard benchmark for long-horizon, language-conditioned manipulation.
  • Robosuite (Cluttered Nut Assembly): A challenging simulation environment requiring precise manipulation in crowded scenes.

Key Insights and Ideas

  • Latent vs. Pixel Space: Verifying in the DINOv2 latent space is roughly 60x faster than pixel-diffusion models (0.08s vs 5.1s per rollout). It also avoids the "hallucinations" common in generative video models.
  • The Power of the Critic: The VLM is already a decent planner (93.3% plan correctness on simple tasks). The real value of Verify2Act isn't just "fixing" bad plans, but reliably filtering out the small fraction of incorrect ones before the robot moves.
  • Handling Occlusion: By using a temporal history window, the model avoids "amnesia." When the gripper closes around a block, the model remembers the block is still there because it "sees" the history of the scene.
  • Sim-to-Real via Digital Twins: The world model and critic were trained entirely on synthetic data from a digital twin, yet they generalized to real-world camera frames with high accuracy.
Detailed architecture of the V2A-WM training and inference process.

Key Results

  • Real-World Success: Verify2Act achieved an 83.3% task success rate, significantly outperforming the VLM-only baseline (53.3%) and a pixel-diffusion world model (46.7%).
  • Zero Planning Failures: On the robot, 100% of the plans executed by Verify2Act were semantically and physically correct. Every single failure was due to low-level execution slips (e.g., a block dropping), not a bad plan.
  • Efficiency: In the CALVIN simulation, Verify2Act reached a 75% success rate on 5-task chains while requiring 30x fewer calls to the expensive VLM compared to reflective diffusion-based methods.
  • Robustness: Even without the real2sim alignment step, the model ranked the correct plan first in 71% of real-world scenes, proving the latent features are robust to visual domain shifts.
A real-world example where the critic rejects an incorrect VLM plan and accepts a correct re-plan.

Limitations and Future Work

  • Task Complexity: The current evaluation is limited to tabletop block manipulation. Future work is needed to test broader geometries and contact-rich tasks (like opening drawers or using tools).
  • Alignment Overhead: The real2sim alignment step, while helpful, adds several seconds of processing time per frame.
  • Dynamic Backgrounds: The "background-sparsity mask" used to keep the world model stable assumes a static camera and background. This might fail if the camera moves or if there are other moving objects in the scene.
  • Low-Level Skills: Since the framework is "policy-agnostic," it still relies on the quality of the underlying low-level controllers. Improving these controllers would likely push the 83.3% success rate even higher.

Summary updated

Abstract

Vision-Language Models (VLMs) can propose multi-step manipulation plans, but they have no grounded model of how a scene will change, and the pixel-level world models that could check those plans are slow and drift over long horizons. We present Verify2Act, a modular verification framework that separates semantic action proposal from latent physical verification. A VLM proposes candidate plans, a latent world model (V2A-WM) imagines each of them in the DINOv2 feature space, and a dual-head critic checks the imagined rollouts for temporal consistency and goal satisfaction before anything is executed. V2A-WM extends recent flow-matching world models with cross-attention action grounding and a causal temporal history window. We deploy the full system on a low-cost robot arm, training the world model and critic only on data generated in a digital twin. Across six language-conditioned tasks, Verify2Act reaches 83.3\% task success, compared with 61.7\% for a latent world model without our architectural changes, 53.3\% for the VLM alone, and 46.7\% for a pixel-diffusion world model. It verified every plan it executed and incurred no planning-induced failures: all of its failed episodes were caused by low-level grasp or placement slips. In simulation, Verify2Act obtains the highest five-task chain success on CALVIN among the compared baselines with 30$\times$ fewer reflective VLM calls than pixel-diffusion verification, and the highest success on cluttered nut assembly.
BibTeX
@misc{eze2026verify,
  author = {Chrisantus Eze and Christopher Crick},
  title = {{Verify2Act: Critic-Guided Latent World Models for Verifying Language-Conditioned Manipulation Plans}},
  year = {2026},
  eprint = {2611.25243},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25243},
}
Figure