TL;DR
This paper introduces VLM-GIST, a novel framework that integrates Vision-Language Models (VLMs) with open-vocabulary object detection and video segmentation to enable robots to understand and track objects in complex, dynamic environments without prior training. This approach allows robots to extract task-relevant attributes and segment objects in real-time, bridging the gap between powerful VLM understanding and practical robotic perception.
What Problem Does This Paper Address?
Robots operating in real-world environments need to perceive and interact with a vast array of objects, often in unpredictable situations. Traditional object detection methods are limited by fixed vocabularies (closed-vocabulary) and struggle with novel objects. While recent Vision-Language Models (VLMs) like GPT and Gemini can describe complex scenes and even identify objects without prior training (open-vocabulary), they have several limitations for robotics:
- Slow Inference: Generating detailed descriptions takes too long for real-time robotic tasks.
- Unstructured Output: VLM descriptions are natural language, not easily parsed by robots for action.
- Lack of Grounding: Descriptions aren't directly linked to specific object locations in the image.
- Need for Dense Understanding: Many robotic tasks require precise pixel-level segmentation, not just bounding boxes.
This paper aims to bridge this gap by creating a system that leverages the descriptive power of VLMs while overcoming their speed and grounding limitations, enabling robots to perform tasks like instance segmentation and tracking in dynamic, open-world scenarios.
Methods
The core of VLM-GIST is a pipeline that combines several components:
- VLM-Powered Structured Description: A VLM is prompted to generate a "structured description" of visible objects. This description is formatted as JSON, containing a
object_nameand adescriptionfor each object. Thedescriptionis specifically crafted to be used as a prompt for an open-vocabulary detector. The authors also introduce a "decoupled attribution" mechanism to mitigate biases in VLM responses for specific task-relevant attributes. - Open-Vocabulary Detection: The generated descriptions are fed into an open-vocabulary object detector (specifically MM-Grounding-DINO is used). This detector outputs bounding boxes for the identified objects. An "Over-Detect-Factor" (ODF) is used to manage potential multiple detections per object instance.
- Instance Segmentation and Tracking: The bounding boxes are then used to prompt a video segmentation model (SAM 2) to generate precise segmentation masks. This model then tracks these instances across the video stream, providing high-frequency segmentation masks with minimal computational overhead.
- Validation and Error Correction (Optional): An optional step uses another VLM prompt to validate the generated object-mask pairings, further improving precision at the cost of some recall.
This approach allows for a low-frequency update mechanism (driven by the VLM and detector) to initialize or update the tracker, while the tracker itself runs at high frequency on the video stream.
Datasets
- Custom Dataset: The authors introduce a new dataset featuring diverse, non-standard objects in domestic environments, designed to be challenging for manipulation tasks. It contains 64 images with detailed, instance-specific natural language descriptions.
- COCO minival: A standard dataset used for evaluating the performance of the open-vocabulary detectors.
Key Insights and Ideas
- VLMs as "Oracles": VLMs can be effectively used to automate the creation of prompts for object detectors, acting as an "oracle" that provides the necessary semantic information without requiring manual prompt engineering.
- Structured Descriptions are Key: The success hinges on generating machine-readable, structured descriptions from VLMs, rather than just free-form text.
- Decoupling Task Relevance: Directly asking VLMs for task-specific attributes can lead to biases. Decoupling this by first generating general descriptions and then separately identifying task-relevant objects improves robustness.
- Real-time Tracking with Segmentation: Combining a lightweight video segmentation model with VLM-driven updates allows for high-frequency, dense object tracking, crucial for robotic interaction.
- Generalization Across Models: The framework is designed to be model-agnostic, allowing easy swapping of VLMs and detectors, demonstrating robust performance across different model capabilities.
Key Results
- Robotics Performance: The method was successfully used in a winning robot demonstration at RoboCup@Home 2024, performing complex tasks like identifying and grasping ingredients based on recipes.
- Quantitative Evaluation: On a mobile manipulator, the system achieved a 67% success rate on challenging tasks, with failures primarily attributed to planning, navigation, or manipulation, not the vision system itself.
- Detector Performance: MM-Grounding-DINO showed strong performance, especially when prompted with more information. The authors' method effectively approximates an "instance-aware oracle" for detectors.
- Speed: While VLM inference can be slow (e.g., 2.5s for a quantized model on complex scenes), the overall pipeline is designed for real-time tracking, with segmentation and detection being much faster.
Limitations and Future Work
- VLM Latency: The primary bottleneck remains the VLM's inference time for generating structured descriptions, though this is mitigated by using it for low-frequency updates.
- Dataset Bias: Performance can be affected by the richness of the VLM's world knowledge and the quality of the training data for the detectors.
- Future Work: The authors suggest that their framework can be used to automatically annotate datasets, which could further improve the performance of open-vocabulary detectors and VLMs. They also highlight the potential for using such datasets to improve robotic perception and manipulation capabilities.
Summary updated