Introduction
Achieving open-vocabulary robotic manipulation—where a robot can perform tasks based on arbitrary natural language instructions—remains a significant challenge. Traditional systems often struggle with the gap between high-level semantic understanding and the low-level physical dynamics required for complex tasks. KUDA (Keypoints to Unify Dynamics and Visual Prompting) bridges this gap by using keypoints as a universal representation that connects Vision-Language Models (VLMs) with physics-based planning.
The Challenge: Dynamics vs. Semantics
Current open-vocabulary systems typically face two major hurdles:
- Ignoring Dynamics: Many systems use VLMs to select a single interaction point (e.g., a click or a push). While effective for rigid objects, this approach fails for deformable objects like ropes or multi-object systems like piles of beans, where the physics are complex and non-linear.
- Rigid Goal Specification: Model-based planning requires precise mathematical target states. Extracting these from vague natural language (e.g., "straighten the rope") is notoriously difficult.
KUDA addresses these issues by adopting keypoints as a "middle ground" representation. Keypoints are abstract enough for a VLM to reason about spatially, yet precise enough to serve as coordinates for a physics-based cost function.
The KUDA Framework
The KUDA pipeline operates through a multi-stage process that integrates visual perception, language reasoning, and neural dynamics:
- Keypoint Sampling & Visual Prompting: The system utilizes Segment Anything (SAM) to identify objects and Farthest Point Sampling (FPS) to distribute keypoints across them. These keypoints are labeled on the RGB image, creating a "visual prompt" that allows the VLM to refer to specific parts of an object.
- VLM as a Geometric Coder: KUDA queries a VLM (such as GPT-4o) with the labeled image. Instead of requesting a direct action, it asks the VLM to generate Python code defining the target positions of the keypoints relative to reference markers.
- Neural Dynamics & MPPI Planning: Learned neural dynamics models (Graph Neural Networks) predict object movement. The system then employs Model Predictive Path Integral (MPPI) control to find the optimal action sequence to reach the VLM-defined targets.
- Closed-Loop Control: A two-level loop ensures accuracy: a low-level loop handles real-time physics adjustments, while a high-level loop re-queries the VLM to correct strategic drift.
Experimental Evaluation
KUDA was evaluated in real-world tabletop environments across four object categories: ropes, rigid cubes, granular piles (coffee beans), and T-shaped blocks. The system was tested on 60 trials involving six distinct tasks, such as "Collect all coffee beans into the square" and "Move the T-shape to make the word 'ROBOT'."
Key Results
KUDA demonstrated a significant performance leap over state-of-the-art baselines like MOKA and VoxPoser:
- Overall Success Rate: KUDA achieved an 80.0% success rate, vastly outperforming baselines which averaged only 13.3%.
- Deformable and Granular Mastery: In tasks involving rope straightening and granular collection—where baselines frequently failed (0-20% success)—KUDA maintained high performance (60-100%).
- Semantic Grounding: The system successfully handled nuanced instructions, such as distinguishing between physical colors and text labels (e.g., "the pink sticker" vs. "the sticker that writes 'pink'").
Conclusion and Future Work
KUDA proves that keypoints are a powerful unifying representation for robotic manipulation. By combining the reasoning capabilities of VLMs with the precision of neural dynamics, it enables robots to handle complex, non-rigid objects through simple natural language commands. Future work will focus on addressing perception bottlenecks in dense environments and expanding the system to 3D spatial relationships and occluded objects.
Summary updated