Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation

Jianzong Wu, Xiangtai Li, Henghui Ding, Xia Li 0005, Guangliang Cheng, Yunhai Tong, Chen Change Loy

Abstract

Open-vocabulary instance segmentation (OVIS) seeks to identify and segment objects beyond a fixed set of training categories by leveraging natural language descriptions. However, standard approaches that ground every word in a caption to image regions often introduce noise, as non-object words (e.g., adjectives, verbs) lack clear spatial boundaries. We introduce the Caption Grounding and Generation (CGG) framework, which addresses this by selectively grounding only object nouns while utilizing a caption generation head to preserve contextual information. Our method achieves a significant performance boost on the COCO benchmark, reaching 28.4% mAP on novel classes—a 6.8% improvement over previous state-of-the-art models without the need for additional external data.

Introduction

Traditional instance segmentation models are restricted to a closed set of categories seen during training. OVIS aims to overcome this by using image-caption pairs to learn a broader range of visual concepts. A common strategy is caption grounding, which maps words to image regions. However, this approach is often hindered by the inherent nature of captions. Words like "messy" or "beautiful" do not correspond to specific segmentable objects, leading to noisy supervision. Furthermore, by focusing solely on grounding, models may ignore the rich contextual cues provided by verbs and adjectives that could help identify novel objects.

Comparison of grounding strategies: grounding all words vs. focusing on object nouns.

The CGG Framework

The proposed CGG framework, built upon the Mask2Former architecture, decouples the learning process into two specialized paths:

  • Selective Grounding: The model extracts object nouns from captions and aligns them with multi-modal embeddings. This ensures that spatial supervision is derived from the most reliable linguistic anchors.
  • Caption Generation: A lightweight Transformer-based head reconstructs the full original caption from visual features. This forces the model to encode global context and relationships (e.g., "riding", "wooden") into its internal representations without requiring explicit spatial localization for every word.
The CGG architecture featuring dual heads for selective grounding and caption generation.

By training these components jointly, the model learns precise object boundaries through noun grounding while gaining a deeper understanding of the scene through generation. The caption generation head is only used during training, adding negligible overhead (0.8% GFlops) and ensuring inference remains efficient.

Experimental Evaluation

CGG was evaluated on the COCO dataset for both OVIS and Open Set Panoptic Segmentation (OSPS). In the OVIS task, the model achieved 28.4% mAP on novel classes, outperforming the previous state-of-the-art (XPM) by 6.8%. In the OSPS task, it showed a 15% improvement in Panoptic Quality (PQ) for unknown categories. Notably, these results were obtained using only the standard COCO-captions training set, demonstrating high data efficiency. Zero-shot transfer tests on the ADE20K dataset further confirmed the model's ability to generalize to unseen categories not present in the COCO training set.

Conclusion

The CGG framework demonstrates that selectively using captions for grounding and generation can significantly enhance open-vocabulary segmentation. By treating nouns as spatial anchors and other words as contextual supervisors, the model avoids the pitfalls of noisy grounding. While current results are promising, future work will explore scaling the framework to larger datasets like LVIS and integrating large-scale vision-language models like CLIP for further distillation.

Summary updated

Abstract

In this work, we focus on open vocabulary instance segmentation to expand a segmentation model to classify and segment instance-level novel categories. Previous approaches have relied on massive caption datasets and complex pipelines to establish one-to-one mappings between image regions and words in captions. However, such methods build noisy supervision by matching non-visible words to image regions, such as adjectives and verbs. Meanwhile, context words are also important for inferring the existence of novel objects as they show high inter-correlations with novel categories. To overcome these limitations, we devise a joint \textbf{Caption Grounding and Generation (CGG)} framework, which incorporates a novel grounding loss that only focuses on matching object nouns to improve learning efficiency. We also introduce a caption generation head that enables additional supervision and contextual modeling as a complementation to the grounding loss. Our analysis and results demonstrate that grounding and generation components complement each other, significantly enhancing the segmentation performance for novel classes. Experiments on the COCO dataset with two settings: Open Vocabulary Instance Segmentation (OVIS) and Open Set Panoptic Segmentation (OSPS) demonstrate the superiority of the CGG. Specifically, CGG achieves a substantial improvement of 6.8% mAP for novel classes without extra data on the OVIS task and 15% PQ improvements for novel classes on the OSPS benchmark.
BibTeX
@inproceedings{wu2023betrayed,
  author = {Jianzong Wu and Xiangtai Li and Henghui Ding and Xia Li and Guangliang Cheng and Yunhai Tong and Chen Change Loy},
  title = {{Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure