Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Tracking

Bastian Pätzold, Jan Nogga, Sven Behnke

TL;DR

This paper introduces VLM-GIST, a novel framework that integrates Vision-Language Models (VLMs) with open-vocabulary object detection and video segmentation to enable robots to understand and track objects in complex, dynamic environments without prior training. This approach allows robots to extract task-relevant attributes and segment objects in real-time, bridging the gap between powerful VLM understanding and practical robotic perception.

What Problem Does This Paper Address?

Robots operating in real-world environments need to perceive and interact with a vast array of objects, often in unpredictable situations. Traditional object detection methods are limited by fixed vocabularies (closed-vocabulary) and struggle with novel objects. While recent Vision-Language Models (VLMs) like GPT and Gemini can describe complex scenes and even identify objects without prior training (open-vocabulary), they have several limitations for robotics:

  • Slow Inference: Generating detailed descriptions takes too long for real-time robotic tasks.
  • Unstructured Output: VLM descriptions are natural language, not easily parsed by robots for action.
  • Lack of Grounding: Descriptions aren't directly linked to specific object locations in the image.
  • Need for Dense Understanding: Many robotic tasks require precise pixel-level segmentation, not just bounding boxes.

This paper aims to bridge this gap by creating a system that leverages the descriptive power of VLMs while overcoming their speed and grounding limitations, enabling robots to perform tasks like instance segmentation and tracking in dynamic, open-world scenarios.

Methods

The core of VLM-GIST is a pipeline that combines several components:

  1. VLM-Powered Structured Description: A VLM is prompted to generate a "structured description" of visible objects. This description is formatted as JSON, containing a object_name and a description for each object. The description is specifically crafted to be used as a prompt for an open-vocabulary detector. The authors also introduce a "decoupled attribution" mechanism to mitigate biases in VLM responses for specific task-relevant attributes.
  2. Open-Vocabulary Detection: The generated descriptions are fed into an open-vocabulary object detector (specifically MM-Grounding-DINO is used). This detector outputs bounding boxes for the identified objects. An "Over-Detect-Factor" (ODF) is used to manage potential multiple detections per object instance.
  3. Instance Segmentation and Tracking: The bounding boxes are then used to prompt a video segmentation model (SAM 2) to generate precise segmentation masks. This model then tracks these instances across the video stream, providing high-frequency segmentation masks with minimal computational overhead.
  4. Validation and Error Correction (Optional): An optional step uses another VLM prompt to validate the generated object-mask pairings, further improving precision at the cost of some recall.

This approach allows for a low-frequency update mechanism (driven by the VLM and detector) to initialize or update the tracker, while the tracker itself runs at high frequency on the video stream.

Datasets

  • Custom Dataset: The authors introduce a new dataset featuring diverse, non-standard objects in domestic environments, designed to be challenging for manipulation tasks. It contains 64 images with detailed, instance-specific natural language descriptions.
  • COCO minival: A standard dataset used for evaluating the performance of the open-vocabulary detectors.

Key Insights and Ideas

  • VLMs as "Oracles": VLMs can be effectively used to automate the creation of prompts for object detectors, acting as an "oracle" that provides the necessary semantic information without requiring manual prompt engineering.
  • Structured Descriptions are Key: The success hinges on generating machine-readable, structured descriptions from VLMs, rather than just free-form text.
  • Decoupling Task Relevance: Directly asking VLMs for task-specific attributes can lead to biases. Decoupling this by first generating general descriptions and then separately identifying task-relevant objects improves robustness.
  • Real-time Tracking with Segmentation: Combining a lightweight video segmentation model with VLM-driven updates allows for high-frequency, dense object tracking, crucial for robotic interaction.
  • Generalization Across Models: The framework is designed to be model-agnostic, allowing easy swapping of VLMs and detectors, demonstrating robust performance across different model capabilities.

Key Results

  • Robotics Performance: The method was successfully used in a winning robot demonstration at RoboCup@Home 2024, performing complex tasks like identifying and grasping ingredients based on recipes.
  • Quantitative Evaluation: On a mobile manipulator, the system achieved a 67% success rate on challenging tasks, with failures primarily attributed to planning, navigation, or manipulation, not the vision system itself.
  • Detector Performance: MM-Grounding-DINO showed strong performance, especially when prompted with more information. The authors' method effectively approximates an "instance-aware oracle" for detectors.
  • Speed: While VLM inference can be slow (e.g., 2.5s for a quantized model on complex scenes), the overall pipeline is designed for real-time tracking, with segmentation and detection being much faster.

Limitations and Future Work

  • VLM Latency: The primary bottleneck remains the VLM's inference time for generating structured descriptions, though this is mitigated by using it for low-frequency updates.
  • Dataset Bias: Performance can be affected by the richness of the VLM's world knowledge and the quality of the training data for the detectors.
  • Future Work: The authors suggest that their framework can be used to automatically annotate datasets, which could further improve the performance of open-vocabulary detectors and VLMs. They also highlight the potential for using such datasets to improve robotic perception and manipulation capabilities.

Summary updated

Abstract

Vision-language models (VLMs) excel in visual understanding but often lack reliable grounding capabilities and actionable inference rates. Integrating them with open-vocabulary object detection (OVD), instance segmentation, and tracking leverages their strengths while mitigating these drawbacks. We utilize VLM-generated structured descriptions to identify visible object instances, collect application-relevant attributes, and inform an open-vocabulary detector to extract corresponding bounding boxes that are passed to a video segmentation model providing segmentation masks and tracking. Once initialized, this model directly extracts segmentation masks, processing image streams in real time with minimal computational overhead. Tracks can be updated online as needed by generating new structured descriptions and detections. This combines the descriptive power of VLMs with the grounding capability of OVD and the pixel-level understanding and speed of video segmentation. Our evaluation across datasets and robotics platforms demonstrates the broad applicability of this approach, showcasing its ability to extract task-specific attributes from non-standard objects in dynamic environments. Code, data, videos, and benchmarks are available at https://vlm-gist.github.io
BibTeX
@article{patzold2025leveraging,
  author = {Bastian Pätzold and Jan Nogga and Sven Behnke},
  title = {{Leveraging Vision-Language Models for Open-Vocabulary Instance Segmentation and Tracking}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2025},
}
Figure