Text and Click inputs for unambiguous open vocabulary instance segmentation

Vighnesh Birodkar, Jonathan Huang, Meera Hahn, Irfan Essa, Nikolai Warner

Introduction

Instance segmentation—the task of assigning pixel-level labels to specific object instances—is a cornerstone of computer vision. However, traditional models often struggle with two primary constraints: a fixed vocabulary and the inherent ambiguity of user intent in interactive settings. This paper presents a novel framework that addresses these challenges by combining a single user click with a natural language prompt. By leveraging the open-vocabulary capabilities of CLIP, the proposed model achieves robust segmentation of unseen objects and resolves the "part-whole" ambiguity that plagues click-only systems.

The Challenge: "Which" vs. "What"

Interactive segmentation typically relies on user clicks to guide the model. However, a single click is spatially ambiguous. For example, a click on a person's necktie could reasonably be interpreted as a request to segment the tie, the shirt, or the person. Conversely, text-only prompts (e.g., "car") are semantically clear but spatially ambiguous when an image contains multiple instances of the same category.

The authors propose a "Text + Click" paradigm to resolve this. In this setup, the text prompt defines the category (the "what"), while the click specifies the instance (the "which").

Methodology: The 5-Channel Framework

The core of the approach is a modification of the standard DeepLabV3+ architecture. Instead of processing a standard 3-channel RGB image, the model utilizes a 5-channel input representation:

  1. RGB Channels (1-3): The original image data.
  2. Click Map (Channel 4): A spatial map generated from the user's foreground click using a Euclidean distance transform. This provides a localizing signal for the specific instance.
  3. Text Saliency Map (Channel 5): A semantic heatmap generated via MaskCLIP (a modified version of OpenAI’s CLIP). This map highlights regions in the image that correspond to the natural language prompt.

By offloading the semantic "knowledge" to a pre-trained CLIP model, the segmentation backbone can be trained in a class-agnostic manner. This allows the network to focus on refining object boundaries and binary mask generation (foreground vs. background) rather than learning a fixed set of categories.

Experimental Evaluation

The model's performance was evaluated across four diverse benchmarks: refCOCO, MS COCO, Pascal VOC, and OpenImages. The primary focus was on "Zero-Shot Segmentation," testing the model's ability to generalize to classes not seen during the training of the segmentation backbone.

Key Findings

  • Zero-Shot Generalization: On the COCO dataset, the inclusion of the text-saliency map resulted in a 6.98 mIoU improvement for "unseen" classes compared to a baseline that used only click inputs.
  • Outperforming Large-Scale Models: Despite being trained on significantly smaller datasets, the proposed model outperformed Meta’s Segment Anything Model (SAM) in specific interactive scenarios. On the COCO dataset, using a single click and text prompt, the model achieved 47.17 mIoU, compared to SAM’s 39.31 mIoU.
  • Robustness to Distractors: The model demonstrated superior performance even in complex scenes containing multiple objects of the same class, as the text saliency map effectively filtered out irrelevant regions while the click localized the target instance.

Discussion and Limitations

While the "Text + Click" approach offers significant advantages, the authors identified specific failure modes:

  • Whole-Object Bias: The model occasionally defaults to segmenting a larger, more common object (e.g., a "car") even when the text prompt and click specify a sub-part (e.g., a "wheel"). This is likely due to the prevalence of whole-object examples in the training data.
  • Saliency Dependence: The quality of the final segmentation is heavily dependent on the initial saliency map produced by MaskCLIP. If the prompt is highly obscure or the CLIP embedding is noisy, the segmentation network may fail to produce an accurate mask.

Conclusion

This research demonstrates that combining linguistic and spatial cues is a powerful strategy for open-vocabulary instance segmentation. By integrating CLIP-based saliency maps with user clicks, the model effectively resolves ambiguity and generalizes to unseen categories without the need for massive, category-specific training sets. Future work may focus on automating category detection to further streamline the interactive process.

Summary updated

Abstract

Abstract
BibTeX
@inproceedings{birodkar2023text,
  author = {Vighnesh Birodkar and Jonathan Huang and Meera Hahn and Irfan Essa and Nikolai Warner},
  title = {{Text and Click inputs for unambiguous open vocabulary instance segmentation}},
  booktitle = {BMVC},
  year = {2023},
  url = {https://openreview.net/forum?id=xV0FK6pALH},
}
Figure