TL;DR
This paper introduces a zero-shot framework that bridges the gap between high-level language instructions and low-level physical execution for bimanual robots. By integrating geometric constraints like accessibility and reachability directly into the planning process and employing a hierarchical validation system, the robot can navigate cluttered environments and recover from failures far more efficiently than standard vision-language models.
What Problem Does This Paper Address?
While Large Language Models (LLMs) and Vision-Language Models (VLMs) have become remarkably good at high-level reasoning—like understanding that "cleaning the table" involves moving a cup—they often struggle with the "how." In a cluttered environment, a robot doesn't just need to know which object to move; it needs to know if there is enough physical clearance for its gripper to fit between two boxes, or if its arm's kinematics can actually reach a specific side of a bottle.
For bimanual (two-armed) robots, this challenge is doubled. Coordinating two manipulators requires reasoning about mutual obstruction and kinematic limits in real-time. Most existing systems either reason at a high semantic level (ignoring physics) or learn end-to-end behaviors (losing the ability to explain failures). When these systems fail, they often "replan from scratch," a computationally expensive process that ignores the progress already made.
Methods
The authors propose a modular, zero-shot pipeline that grounds symbolic planning in physical reality. The process follows four main stages:
- Structured Scene Representation: The system uses a VLM to identify objects and their semantic relations, then combines this with 6D pose estimation (using FoundationPose). Crucially, it annotates the scene graph with accessibility (is there enough free space for a gripper?) and reachability (can the robot's inverse kinematics solve for that pose?).
- Execution-Aware Planning: Instead of a generic "pick" command, the planner generates actions specified by the target object, the specific arm to use, the contact mode, and the exact side of the object to grasp.
- Parallel Scheduling: A dedicated module analyzes the sequential plan and groups compatible actions into parallel stages. For example, if the robot needs to move two bottles, the scheduler determines if it can safely use both arms simultaneously to save time.
- Hierarchical Validation & Recovery: This is the "closed-loop" part of the system. Each stage has explicit pre- and post-conditions. If a VLM-based validator detects a failure, it doesn't just restart. It follows a local-to-global recovery policy:
- Repeat: Try the exact same action again (useful for minor slips).
- Modify: Change a specific parameter, like trying a different grasp side, based on the failure feedback.
- Replan: Only if the previous two fail does the system trigger a full global replanning.
Datasets
As a zero-shot framework, the system does not require task-specific training data. Instead, it was evaluated on the CENTAURO humanoid robot across nine distinct tabletop manipulation scenarios designed to test different capabilities:
- Planning: Stacked Boxes, Boxes Aside, Bottles on Box, Fragile Box, Long Horizon.
- Adaptation/Recovery: Scene Update, Bottle Failure, Push Failure, Stacked Boxes w/ Failure.
Key Insights and Ideas
- The "Lateral Obstacle" Problem: Standard VLM planners often recognize when an object is on top of a target but fail to realize when objects placed side-by-side prevent a gripper from entering. By explicitly encoding "side-level accessibility," this framework solves scenarios where baselines fail completely.
- Validation as Interpretation: By treating validation as a three-way outcome (Satisfied, Uncertain, Violated) rather than a binary success/failure, the robot can proactively seek more visual information when it's "unsure" before committing to a recovery strategy.
- Efficiency through Locality: The hierarchical recovery strategy treats the plan as a series of "independently verifiable properties." This allows the robot to fix a "broken" step without throwing away the entire successful sequence that led up to it.
Key Results
The framework was tested against two baselines: one that was not execution-aware and one that used simple binary success/failure validation.
- Success Rate (SR): In the "Boxes Aside" scenario (where objects are laterally cluttered), the proposed framework achieved an 85% success rate, while the non-execution-aware baseline failed every single time (0% SR).
- Query Efficiency: In failure scenarios like "Bottle Failure," the hierarchical validator reduced the average number of VLM queries from 10 to 8.1 per execution, as it could often "Repeat" or "Modify" rather than "Replan."
- Bimanual Speedup: The parallel scheduler reduced execution time by 40% in the "Bottles on Box" scenario by successfully coordinating both arms.
- VLM Backbone: The system showed robustness across different models; while Gemini Robotics-ER 1.6 was the primary backbone, GPT-o3 achieved a comparable 80% SR, though with significantly higher latency.
Limitations and Future Work
- Fixed Base: Currently, the robot evaluates reachability from a stationary position. If an object is out of reach, it simply discards that option. Future iterations aim to integrate base motion into the planning.
- Geometric Assumptions: The system currently approximates objects as oriented bounding boxes. This works for many household items but fails for objects with complex affordances, like a mug that must be grasped by a specific handle.
- Contact-Rich Behaviors: The authors suggest that replacing standard inverse kinematics with learned manipulation policies could allow for more adaptive, "contact-rich" behaviors (like sliding an object) that are difficult to model geometrically.
Summary updated