ICPC: Instance-Conditioned Prompting with Contrastive Learning for Semantic Segmentation

Chaohui Yu, Qiang Zhou, Zhibin Wang, Fan WangarXiv 2023 · DOI · arXiv

Introduction

Semantic segmentation—the task of assigning a category label to every pixel in an image—has traditionally relied on backbones pre-trained on ImageNet. Recently, the field has shifted toward leveraging CLIP (Contrastive Language-Image Pre-training). CLIP’s dual-encoder architecture provides a rich, multimodal understanding of visual and textual data, which is highly beneficial for dense prediction tasks.

However, current CLIP-based segmentation methods, such as DenseCLIP, face two primary limitations. First, they utilize static prompting, where a single learnable text prompt is applied to every image regardless of its unique context. Second, they often rely on simple per-pixel classification losses, which provide weak alignment supervision and fail to fully exploit CLIP’s contrastive potential. To address these issues, we present ICPC (Instance-Conditioned Prompting with Contrastive Learning).

The ICPC Framework

The core of the ICPC framework is the transition from static to dynamic prompting. Instead of a fixed learnable context, ICPC introduces Instance-Conditioned Prompting (ICP).

The ICPC framework architecture, highlighting the instance-conditioned prompting and multi-scale alignment modules.

As illustrated in the architecture above, the model generates a dynamic vector ($I$) conditioned on the global features of the input image via a projector. This vector is appended to the learnable context, allowing the text encoder to adapt category embeddings to the specific visual nuances of the scene. Furthermore, to handle objects of varying sizes, the framework performs Multi-scale Vision-Text Alignment across four feature resolutions (1/32 to 1/4) using lightweight transposed convolutions.

Align-Guided Contrastive Learning

To refine the relationship between visual features and text embeddings, ICPC introduces Align-Guided Contrastive Learning (AGCL). Unlike standard classification losses, AGCL enforces a contrastive objective across the mini-batch, pulling "positive" pixel-text pairs together while pushing "negative" pairs apart.

A key component of this strategy is an easy-to-hard sampling mechanism. In the early stages of training, the model prioritizes "easy" samples (those already correctly classified) to establish a stable baseline. As training progresses, the focus shifts toward "hard" samples (misclassified pixels) to sharpen decision boundaries and improve segmentation accuracy in complex regions. This bidirectional pressure ensures that the text embeddings and visual features are tightly coupled in the shared latent space.

Results and Benchmarks

ICPC was evaluated on three major semantic segmentation benchmarks: ADE20K, COCO-Stuff10k, and the high-category ADE20K-Full (847 classes).

Performance comparison (mIoU %) of ICPC against baseline and state-of-the-art methods across three major benchmarks.

As shown in the performance charts, ICPC consistently outperforms previous state-of-the-art methods. On ADE20K with a ResNet-50 backbone, ICPC achieved 45.2% mIoU, representing a +1.71% improvement over DenseCLIP. The gains were even more significant on the challenging ADE20K-Full dataset, with improvements ranging from +1.41% to +1.68% mIoU across various backbones.

Beyond standard benchmarks, the model demonstrated robust zero-shot transfer capabilities. When trained on COCO-Stuff and tested on datasets like Cityscapes and iSAID without further tuning, ICPC maintained superior performance, suggesting that the learned vision-text alignment generalizes well to unseen domains.

Discussion and Limitations

The primary advantage of ICPC lies in its ability to produce context-aware category embeddings. By allowing the image to influence the text prompt, the model can better distinguish between similar categories in different environments (e.g., a "sink" in a kitchen versus a bathroom). T-SNE visualizations confirm that ICPC produces much tighter clusters in the embedding space compared to static methods.

However, this dynamic approach introduces a trade-off in inference speed. Because the text prompt is image-dependent, the text encoder must be executed for every new input. In contrast, static methods can pre-compute and cache text embeddings. While the computational overhead is relatively small (3–5% increase in FLOPs), the requirement for per-image text encoding may limit its use in real-time applications. Future research will focus on more efficient dynamic prompting strategies that maintain performance while reducing the frequency of text encoder passes.

Summary updated

Abstract

Modern supervised semantic segmentation methods are usually finetuned based on the supervised or self-supervised models pre-trained on ImageNet. Recent work shows that transferring the knowledge from CLIP to semantic segmentation via prompt learning can achieve promising performance. The performance boost comes from the feature enhancement with multimodal alignment, i.e., the dot product between vision and text embeddings. However, how to improve the multimodal alignment for better transfer performance in dense tasks remains underexplored. In this work, we focus on improving the quality of vision-text alignment from two aspects of prompting design and loss function, and present an instance-conditioned prompting with contrastive learning (ICPC) framework. First, compared with the static prompt designs, we reveal that dynamic prompting conditioned on image content can more efficiently utilize the text encoder for complex dense tasks. Second, we propose an align-guided contrastive loss to refine the alignment of vision and text embeddings. We further propose lightweight multi-scale alignment for better performance. Extensive experiments on three large-scale datasets (ADE20K, COCO-Stuff10k, and ADE20K-Full) demonstrate that ICPC brings consistent improvements across diverse backbones. Taking ResNet-50 as an example, ICPC outperforms the state-of-the-art counterpart by 1.71%, 1.05%, and 1.41% mIoU on the three datasets, respectively.
BibTeX
@misc{yu2023icpc,
  author = {Chaohui Yu and Qiang Zhou and Zhibin Wang and Fan Wang},
  title = {{ICPC: Instance-Conditioned Prompting with Contrastive Learning for Semantic Segmentation}},
  year = {2023},
  eprint = {2308.07078},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2308.07078},
}
Figure