A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future

Chaoyang Zhu, Long Chen

Abstract

As the most fundamental tasks of computer vision, object detection and segmentation have made tremendous progress in the deep learning era. Due to the expensive manual labeling, the annotated categories in existing datasets are often small-scale and pre-defined, i.e., state-of-the-art detectors and segmentors fail to generalize beyond the closed-vocabulary. To resolve this limitation, the last few years have witnessed increasing attention toward Open-Vocabulary Detection (OVD) and Segmentation (OVS). In this survey, we provide a comprehensive review on the past and recent development of OVD and OVS. To this end, we develop a taxonomy according to the type of task and methodology. We find that the permission and usage of weak supervision signals can well discriminate different methodologies, including: visual-semantic space mapping, novel visual feature synthesis, region-aware training, pseudo-labeling, knowledge distillation-based, and transfer learning-based. The proposed taxonomy is universal across different tasks, covering object detection, semantic/instance/panoptic segmentation, 3D scene and video understanding. In each category, its main principles, key challenges, development routes, strengths, and weaknesses are thoroughly discussed. In addition, we benchmark each task along with the vital components of each method. Finally, several promising directions are provided to stimulate future research.

Introduction: From Closed to Open Vocabularies

Traditional object detection and segmentation systems operate under a "closed-vocabulary" assumption, where models are trained to recognize a fixed, pre-defined set of categories (e.g., the 80 classes in COCO). However, real-world applications often encounter "novel" objects not present in training data. Manual labeling for every possible category is prohibitively expensive; even extensive datasets like LVIS only cover ~1,200 classes.

Open-Vocabulary Detection (OVD) and Segmentation (OVS) aim to bridge this gap, enabling models to localize and classify virtually any object described in natural language. This paradigm shift moves away from fixed learnable classifiers toward aligning visual features with high-dimensional semantic spaces, often leveraging large-scale Vision-Language Models (VLMs) like CLIP.

A Taxonomy of Methodologies

The survey systematizes the field into six primary methodologies, distinguished by their supervision signals and how they bridge the gap between fixed labels and infinite visual concepts.

Taxonomy of OVD/OVS tasks and methodologies, categorizing methods by task type and supervision signal.

1. Visual-Semantic Space Mapping

This "classic" approach learns to project visual features into a fixed semantic space (e.g., Word2Vec or BERT). By mapping visual regions to word embeddings, the model can recognize unseen classes by looking up their corresponding text vectors.

2. Novel Visual Feature Synthesis

A generative approach where semantic embeddings are used to synthesize "fake" visual features for unseen classes. This effectively converts the open-vocabulary problem back into a standard supervised task by augmenting the training set with generated data.

3. Region-Aware Training

Utilizing image-text pairs (captions), these methods "softly" align image regions with words through contrastive losses. While the model doesn't have explicit box-word labels, it learns correspondences from large-scale web data.

4. Pseudo-Labeling

This strategy uses a "teacher" model (typically a pre-trained VLM) to generate explicit labels for novel objects in unannotated images. A "student" model is then trained on these "hard" alignments.

5. Knowledge Distillation

These methods force a detector's backbone to mimic the feature representations of a powerful, frozen VLM like CLIP. This transfers the VLM's broad conceptual knowledge into a spatially-aware detection framework.

6. Transfer Learning

Directly adapts pre-trained VLMs to detection/segmentation tasks using techniques like "prompt tuning" or lightweight adapters. This preserves the VLM's extensive knowledge while fine-tuning it for localization.

Architectural paradigms comparing canonical detectors with open-vocabulary approaches like visual-semantic mapping and region-aware training.

Key Insights and Performance

The "CLIP Catalyst" has been the primary driver of recent progress. The research focus has shifted from learning concepts to extracting and localizing them. While standard VLMs are "spatially blind," OVD/OVS research develops the "eyes" needed for precise localization.

Currently, Knowledge Distillation and Transfer Learning are the most effective strategies, leveraging the massive pre-training of models like CLIP. On benchmarks like LVIS, these methods have achieved significant performance leaps, particularly in Average Precision for rare classes ($AP_r$).

Future Directions

Despite progress, several challenges remain:

  • Overfitting: Models often bias toward common "base" classes seen during training.
  • Background Confusion: Distinguishing novel objects from the background remains difficult.
  • Efficiency: Many state-of-the-art methods are computationally heavy, requiring multiple passes through large VLM encoders.
  • MLLMs: The next frontier involves Multimodal Large Language Models (MLLMs) that can reason about detected objects in complex natural language contexts.

Summary updated

Abstract

As the most fundamental tasks of computer vision, object detection and segmentation have made tremendous progress in the deep learning era. Due to the expensive manual labeling, the annotated categories in existing datasets are often small-scale and pre-defined, i.e., state-of-the-art detectors and segmentors fail to generalize beyond the closed-vocabulary. To resolve this limitation, the last few years have witnessed increasing attention toward Open-Vocabulary Detection (OVD) and Segmentation (OVS). In this survey, we provide a comprehensive review on the past and recent development of OVD and OVS. To this end, we develop a taxonomy according to the type of task and methodology. We find that the permission and usage of weak supervision signals can well discriminate different methodologies, including: visual-semantic space mapping, novel visual feature synthesis, region-aware training, pseudo-labeling, knowledge distillation-based, and transfer learning-based. The proposed taxonomy is universal across different tasks, covering object detection, semantic/instance/panoptic segmentation, 3D scene and video understanding. In each category, its main principles, key challenges, development routes, strengths, and weaknesses are thoroughly discussed. In addition, we benchmark each task along with the vital components of each method. Finally, several promising directions are provided to stimulate future research.
BibTeX
@article{zhu2024survey,
  author = {Chaoyang Zhu and Long Chen},
  title = {{A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2024},
  doi = {10.1109/tpami.2024.3413013},
}
Figure