Abstract
Open-vocabulary instance segmentation (OVIS) seeks to identify and segment objects beyond a fixed set of training categories by leveraging natural language descriptions. However, standard approaches that ground every word in a caption to image regions often introduce noise, as non-object words (e.g., adjectives, verbs) lack clear spatial boundaries. We introduce the Caption Grounding and Generation (CGG) framework, which addresses this by selectively grounding only object nouns while utilizing a caption generation head to preserve contextual information. Our method achieves a significant performance boost on the COCO benchmark, reaching 28.4% mAP on novel classes—a 6.8% improvement over previous state-of-the-art models without the need for additional external data.
Introduction
Traditional instance segmentation models are restricted to a closed set of categories seen during training. OVIS aims to overcome this by using image-caption pairs to learn a broader range of visual concepts. A common strategy is caption grounding, which maps words to image regions. However, this approach is often hindered by the inherent nature of captions. Words like "messy" or "beautiful" do not correspond to specific segmentable objects, leading to noisy supervision. Furthermore, by focusing solely on grounding, models may ignore the rich contextual cues provided by verbs and adjectives that could help identify novel objects.
The CGG Framework
The proposed CGG framework, built upon the Mask2Former architecture, decouples the learning process into two specialized paths:
- Selective Grounding: The model extracts object nouns from captions and aligns them with multi-modal embeddings. This ensures that spatial supervision is derived from the most reliable linguistic anchors.
- Caption Generation: A lightweight Transformer-based head reconstructs the full original caption from visual features. This forces the model to encode global context and relationships (e.g., "riding", "wooden") into its internal representations without requiring explicit spatial localization for every word.
By training these components jointly, the model learns precise object boundaries through noun grounding while gaining a deeper understanding of the scene through generation. The caption generation head is only used during training, adding negligible overhead (0.8% GFlops) and ensuring inference remains efficient.
Experimental Evaluation
CGG was evaluated on the COCO dataset for both OVIS and Open Set Panoptic Segmentation (OSPS). In the OVIS task, the model achieved 28.4% mAP on novel classes, outperforming the previous state-of-the-art (XPM) by 6.8%. In the OSPS task, it showed a 15% improvement in Panoptic Quality (PQ) for unknown categories. Notably, these results were obtained using only the standard COCO-captions training set, demonstrating high data efficiency. Zero-shot transfer tests on the ADE20K dataset further confirmed the model's ability to generalize to unseen categories not present in the COCO training set.
Conclusion
The CGG framework demonstrates that selectively using captions for grounding and generation can significantly enhance open-vocabulary segmentation. By treating nouns as spatial anchors and other words as contextual supervisors, the model avoids the pitfalls of noisy grounding. While current results are promising, future work will explore scaling the framework to larger datasets like LVIS and integrating large-scale vision-language models like CLIP for further distillation.
Summary updated