Bridging VLM and KMP: Enabling Fine-grained robotic manipulation via Semantic Keypoints Representation

Abstract

Autonomous robotic manipulation faces a significant challenge in bridging high-level semantic reasoning with low-level physical precision. While Vision-Language Models (VLMs) excel at understanding complex instructions, they often lack the geometric accuracy required for delicate tasks. Conversely, Movement Primitives (MPs) provide precise trajectory control but struggle with semantic context. This paper introduces VL-MP, a framework that utilizes 3D semantic keypoints as a bridge to translate VLM decisions into precise task parameters. By enhancing Kernelized Movement Primitives (KMP) with local feature preservation, the system achieves high success rates in fine-grained tasks like pouring, even in ambiguous real-world environments.

Introduction

The current landscape of robotic manipulation is divided between two paradigms. VLM-based agents, such as GPT-4o, demonstrate impressive "zero-shot" reasoning capabilities, allowing them to interpret open-ended human instructions. However, these models typically fail when millimeter-level precision is required. On the other hand, traditional Movement Primitives (MPs) are mathematically robust for trajectory generalization but lack the cognitive capacity to handle ambiguous scenes or high-level decision-making.

The authors identify two primary hurdles in unifying these approaches: information distortion during the conversion of high-level decisions to low-level commands, and shape distortion in trajectory generalization. Standard imitation learning methods often "smooth out" critical path details near start and end points, leading to failures in precision-sensitive tasks.

Methods

The VL-MP framework addresses these issues through a three-stage process:

1. Task Decision (VLM): Utilizing GPT-4o and GroundingDINO, the system processes visual and textual inputs. A multi-turn dialogue module is integrated to resolve environmental ambiguities, ensuring the robot correctly identifies the intended target before proceeding.
2. The Bridge Layer (Semantic Keypoints): This core innovation replaces simple coordinate passing with 3D semantic keypoints. Using an HRNet-based network, the system extracts specific interaction points (e.g., a mug's rim), positional points (center), and boundary points (handle). These keypoints provide a robust geometric representation that constrains task parameters, accurately translating the VLM's intent into the robot's physical workspace.

Expand

Source, p. 3

1. Local Feature Enhanced KMP (LFE-KMP): To prevent shape distortion, the authors modified the standard KMP algorithm. LFE-KMP resamples the start and end points of demonstration trajectories. This "anchoring" of local features ensures the robot maintains the intended path shape—such as the specific tilt required for a pour—even when generalizing to new target locations.

Key Insights

A central argument of this work is that 3D semantic keypoints serve as a "universal language" between abstract reasoning and rigid mathematics. They are both human-interpretable and mathematically actionable. Furthermore, the researchers found that by intentionally over-emphasizing local features at trajectory boundaries, they could significantly reduce trajectory divergence, which is the primary cause of failure in fine-grained manipulation.

Expand

Source, p. 4

Key Results

The VL-MP framework was evaluated in both simulated and real-world environments. In ambiguous pouring tasks involving multiple similar targets, VL-MP achieved an 80% success rate, significantly outperforming VLM+Prim (20%) and Diffusion Policy (0%). In handwriting simulations using the GShape dataset, the algorithm demonstrated 92.76% topological similarity, a marked improvement over the 64.95% achieved by standard KMP. Overall, the system showed a 42.82% improvement in shape preservation metrics.

Conclusion and Future Work

VL-MP successfully bridges the gap between high-level reasoning and low-level precision. While currently focused on end-effector trajectories, future work aims to incorporate whole-arm planning to ensure the entire robot configuration avoids obstacles while maintaining precise path-following. Additionally, the team plans to integrate more dynamic environmental constraints into the LFE-KMP framework to handle increasingly complex real-world scenarios.

@misc{zhu2025bridging,
  author = {Junjie Zhu and Huayu Liu and Jin Wang and Bangrong Wen and Kaixiang Huang and Xiaofei Li and Haiyun Zhan and Guodong Lu},
  title = {{Bridging VLM and KMP: Enabling Fine-grained robotic manipulation via Semantic Keypoints Representation}},
  year = {2025},
  eprint = {2503.02748},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.02748},
}