ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation

Ziqin Zhou, Bowen Zhang, Yinjie Lei, Lingqiao Liu, Yifan Liu

TL;DR

ZegCLIP introduces a simple and efficient one-stage framework for zero-shot semantic segmentation by directly extending CLIP's image-level classification power to the pixel level. By incorporating deep prompt tuning, a non-mutually exclusive loss function, and a novel text-image relationship descriptor, it avoids the heavy computational overhead of traditional two-stage proposal methods, achieving state-of-the-art accuracy alongside a 5x inference speedup.

What Problem Does This Paper Address?

Zero-shot semantic segmentation (ZS3) aims to segment image pixels belonging to unseen object categories based solely on their textual descriptions. Recently, large-scale vision-language models like CLIP have been adapted for this task. However, existing state-of-the-art methods typically rely on a clunky, two-stage pipeline:

  1. Generate class-agnostic region proposals.
  2. Crop these regions and pass them to CLIP for image-level classification.

This approach requires two separate image encoding processes, creating massive computational overhead and severely limiting inference speed.

While a one-stage alternative—directly matching CLIP's text embeddings with its patch-level image embeddings—seems intuitive, a naive implementation suffers from severe overfitting. The model learns to excel at recognizing seen classes during training but completely loses its ability to generalize to unseen classes. ZegCLIP tackles this generalization collapse to make one-stage zero-shot segmentation both viable and highly efficient.

Methods

ZegCLIP replaces the complex two-stage pipeline with a streamlined one-stage architecture built on top of a frozen CLIP model and a lightweight transformer decoder. It introduces three simple yet highly effective design choices to preserve CLIP's zero-shot generalization capabilities:

  1. Deep Prompt Tuning (DPT): Instead of freezing the CLIP image encoder entirely (which yields weak representations for dense tasks) or fully fine-tuning it (which destroys its zero-shot capabilities), ZegCLIP injects a small set of learnable prompt tokens into each layer of the Vision Transformer (ViT) backbone while keeping the original weights fixed.
  2. Non-mutually Exclusive Loss (NEL): Standard segmentation models use Softmax and Cross-Entropy loss, assuming classes are mutually exclusive. This miscalibrates logits for unseen classes during inference. ZegCLIP switches to a Sigmoid activation combined with Focal and Dice losses, treating each class prediction independently.
  3. Relationship Descriptor (RD): To prevent the text queries from overfitting seen classes, the authors introduce a descriptor that captures global image context. It computes the Hadamard product between the text embedding ($t$) and the image's global [CLS] token ($g$), concatenating this image-specific relationship descriptor back into the text query before feeding it to the decoder.
Figure 2. Overall of our proposed ZegCLIP. Our method modifies a One-Stage Baseline framework of matching text and patch em- beddings from CLIP to generate semantic masks. The key contribution of our work is three simple-but-effective designs (labeled as the red circles 1,2,3 in the figure). Incorporating these three designs into our proposed one-stage baseline framework can upgrade the poorly performed baseline method to a strong zero-shot segmentation model.

Datasets

The authors evaluate ZegCLIP on three standard benchmarks under both inductive (unseen classes completely hidden during training) and transductive (unseen class names are known, and unlabeled target images are available for self-training) settings:

  • PASCAL VOC 2012: 15 seen classes, 5 unseen classes.
  • COCO-Stuff 164K: A large-scale dataset with 156 seen classes and 15 unseen classes.
  • PASCAL Context: 50 seen classes and 10 unseen classes.

Key Insights and Ideas

  • Fine-Tuning Destroys Open-Vocabulary Knowledge: Fully updating a massive pre-trained model like CLIP on a small, pixel-annotated dataset causes it to rapidly forget its generalized vision-language alignment. Prompt tuning acts as a vital safeguard.
  • The Softmax Trap: Softmax forces a zero-sum competition between classes based on a closed set seen during training. Switching to an independent binary classification setup (NEL) allows unseen classes to be evaluated fairly without being suppressed by dominant seen classes.
  • Global Context Anchors Local Patches: Directly matching text to local patch embeddings leads to disjointed, overfitted masks. Injecting the global text-image relationship descriptor (RD) provides the necessary context to anchor pixel-level predictions to the overall image content.
Figure 1. Quantitative improvements achieved by our proposed designs on VOC dataset. (1)(2) represents our one-stage Base- line model of different versions (Fix or Fine-Tune CLIP image en- coder), while (3)-(5) shows the effectiveness of applying our pro- posed designs, i.e., Deep Prompt Tuning (DPT), Non-mutually Exclusive Loss (NEL), Relationship Descriptor (RD), on base- line model step by step. We highlight that our designs can dramat- ically increase the segmentation performance on unseen classes.

Key Results

  • 5x Speedup: ZegCLIP achieves an inference speedup of approximately 5x compared to prominent two-stage architectures like ZegFormer (e.g., 9.0 FPS vs 1.7 FPS on PASCAL VOC).
  • High Efficiency: It reduces the number of learnable parameters by roughly 76% (~14M vs 60.3M) and slashes computational complexity (FLOPs) by over 90% compared to two-stage baselines.
  • State-of-the-Art Accuracy: On the challenging COCO-Stuff dataset under the inductive setting, ZegCLIP achieves an unseen mIoU of 41.4%, outperforming the previous best method (zsseg at 36.3%) by a substantial margin.
Qualitative segmentation results on the COCO-Stuff dataset comparing the baseline to ZegCLIP.

Limitations and Future Work

  • The Supervised Gap: While ZegCLIP significantly closes the performance gap, a noticeable delta still remains between zero-shot methods and fully supervised models (e.g., on COCO-Stuff, zero-shot unseen mIoU is 41.4% vs. the fully supervised upper bound of 63.2%).
  • Dataset-Specific Prompt Tuning: The optimal number of deep prompt tokens varies significantly across datasets (e.g., 10 tokens for VOC vs. 100 tokens for COCO-Stuff), indicating that manual tuning is required based on dataset scale and complexity.

Summary updated

Abstract

Recently, CLIP has been applied to pixel-level zero-shot learning tasks via a two-stage scheme. The general idea is to first generate class-agnostic region proposals and then feed the cropped proposal regions to CLIP to utilize its image-level zero-shot classification capability. While effective, such a scheme requires two image encoders, one for proposal generation and one for CLIP, leading to a complicated pipeline and high computational cost. In this work, we pursue a simpler-and-efficient one-stage solution that directly extends CLIP's zero-shot prediction capability from image to pixel level. Our investigation starts with a straightforward extension as our baseline that generates semantic masks by comparing the similarity between text and patch embeddings extracted from CLIP. However, such a paradigm could heavily overfit the seen classes and fail to generalize to unseen classes. To handle this issue, we propose three simple-but-effective designs and figure out that they can significantly retain the inherent zero-shot capacity of CLIP and improve pixel-level generalization ability. Incorporating those modifications leads to an efficient zero-shot semantic segmentation system called ZegCLIP. Through extensive experiments on three public benchmarks, ZegCLIP demonstrates superior performance, outperforming the state-of-the-art methods by a large margin under both "inductive" and "transductive" zero-shot settings. In addition, compared with the two-stage method, our one-stage ZegCLIP achieves a speedup of about 5 times faster during inference. We release the code at https://github.com/ZiqinZhou66/ZegCLIP.git.
BibTeX
@inproceedings{zhou2022zegclip,
  author = {Ziqin Zhou and Bowen Zhang and Yinjie Lei and Lingqiao Liu and Yifan Liu},
  title = {{ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
}
Figure