Introduction
Semantic segmentation—the task of assigning a category label to every pixel in an image—has traditionally relied on backbones pre-trained on ImageNet. Recently, the field has shifted toward leveraging CLIP (Contrastive Language-Image Pre-training). CLIP’s dual-encoder architecture provides a rich, multimodal understanding of visual and textual data, which is highly beneficial for dense prediction tasks.
However, current CLIP-based segmentation methods, such as DenseCLIP, face two primary limitations. First, they utilize static prompting, where a single learnable text prompt is applied to every image regardless of its unique context. Second, they often rely on simple per-pixel classification losses, which provide weak alignment supervision and fail to fully exploit CLIP’s contrastive potential. To address these issues, we present ICPC (Instance-Conditioned Prompting with Contrastive Learning).
The ICPC Framework
The core of the ICPC framework is the transition from static to dynamic prompting. Instead of a fixed learnable context, ICPC introduces Instance-Conditioned Prompting (ICP).
As illustrated in the architecture above, the model generates a dynamic vector ($I$) conditioned on the global features of the input image via a projector. This vector is appended to the learnable context, allowing the text encoder to adapt category embeddings to the specific visual nuances of the scene. Furthermore, to handle objects of varying sizes, the framework performs Multi-scale Vision-Text Alignment across four feature resolutions (1/32 to 1/4) using lightweight transposed convolutions.
Align-Guided Contrastive Learning
To refine the relationship between visual features and text embeddings, ICPC introduces Align-Guided Contrastive Learning (AGCL). Unlike standard classification losses, AGCL enforces a contrastive objective across the mini-batch, pulling "positive" pixel-text pairs together while pushing "negative" pairs apart.
A key component of this strategy is an easy-to-hard sampling mechanism. In the early stages of training, the model prioritizes "easy" samples (those already correctly classified) to establish a stable baseline. As training progresses, the focus shifts toward "hard" samples (misclassified pixels) to sharpen decision boundaries and improve segmentation accuracy in complex regions. This bidirectional pressure ensures that the text embeddings and visual features are tightly coupled in the shared latent space.
Results and Benchmarks
ICPC was evaluated on three major semantic segmentation benchmarks: ADE20K, COCO-Stuff10k, and the high-category ADE20K-Full (847 classes).
As shown in the performance charts, ICPC consistently outperforms previous state-of-the-art methods. On ADE20K with a ResNet-50 backbone, ICPC achieved 45.2% mIoU, representing a +1.71% improvement over DenseCLIP. The gains were even more significant on the challenging ADE20K-Full dataset, with improvements ranging from +1.41% to +1.68% mIoU across various backbones.
Beyond standard benchmarks, the model demonstrated robust zero-shot transfer capabilities. When trained on COCO-Stuff and tested on datasets like Cityscapes and iSAID without further tuning, ICPC maintained superior performance, suggesting that the learned vision-text alignment generalizes well to unseen domains.
Discussion and Limitations
The primary advantage of ICPC lies in its ability to produce context-aware category embeddings. By allowing the image to influence the text prompt, the model can better distinguish between similar categories in different environments (e.g., a "sink" in a kitchen versus a bathroom). T-SNE visualizations confirm that ICPC produces much tighter clusters in the embedding space compared to static methods.
However, this dynamic approach introduces a trade-off in inference speed. Because the text prompt is image-dependent, the text encoder must be executed for every new input. In contrast, static methods can pre-compute and cache text embeddings. While the computational overhead is relatively small (3–5% increase in FLOPs), the requirement for per-image text encoding may limit its use in real-time applications. Future research will focus on more efficient dynamic prompting strategies that maintain performance while reducing the frequency of text encoder passes.
Summary updated