Abstract
Weakly Supervised Semantic Segmentation (WSSS) aims to train segmentation models using only image-level labels, significantly reducing annotation costs. While the Contrastive Language-Image Pre-training (CLIP) model has demonstrated exceptional cross-modal alignment, its application in WSSS has largely been limited to generating offline pseudo-labels. We present WeCLIP, a novel single-stage pipeline that utilizes a frozen CLIP model as a direct feature extractor. By training only a lightweight decoder and a dynamic refinement module, WeCLIP achieves state-of-the-art performance on PASCAL VOC and MS COCO benchmarks while maintaining a remarkably low computational footprint (6.2GB VRAM).
Introduction
Semantic segmentation is a fundamental computer vision task that typically requires dense, pixel-level annotations. To mitigate the high cost of data labeling, WSSS leverages "weak" signals like image-level tags. Recent advancements have seen the integration of CLIP, a model trained on 400 million image-text pairs, into WSSS workflows. However, most existing methods treat CLIP as a "teacher" to supervise a separate student network (e.g., ResNet or ViT).
WeCLIP addresses this by using CLIP directly as the backbone. The core challenge is extracting dense pixel-level predictions from CLIP’s high-level semantic features without fine-tuning, which could degrade its pre-trained knowledge. By keeping the backbone frozen, WeCLIP preserves rich semantic relationships and enables training on consumer-grade GPUs.
Methods
The WeCLIP architecture is a single-stage framework composed of four primary modules:
- Frozen CLIP Backbone: Both image and text encoders remain fixed to retain the broad semantic understanding acquired during pre-training.
- Frozen CLIP Feature Decoder: A lightweight transformer-based decoder aggregates features from multiple layers of the CLIP image encoder to generate dense segmentation maps.
- Initial CAM Generation: The model produces an initial Class Activation Map (CAM) using GradCAM and specific text prompts (e.g., "a clear origami {class}") to localize objects.
- CAM Refinement Module (RFM): This module dynamically rectifies the initial, often noisy, pseudo-labels. It utilizes features from the trained decoder to create an affinity map, ensuring that pixels with similar semantic features receive consistent labels.
Key Insights
The Power of Frozen Features
Traditional backbones often require extensive fine-tuning for downstream tasks. In contrast, CLIP’s features are inherently "segmentation-ready." t-SNE visualizations demonstrate that CLIP features naturally form distinct clusters based on object categories even without task-specific training. This inherent structure allows a simple decoder to achieve high accuracy.
Computational Efficiency
By freezing the backbone, WeCLIP only trains approximately 6 million parameters. This efficiency allows the model to run on hardware with as little as 6.2GB of VRAM, a stark contrast to competing single-stage methods like ToCo, which can require over 24GB.
Synergistic Refinement
The decoder and the RFM operate in a mutually beneficial feedback loop. The decoder learns from refined pseudo-labels, and as its feature representation improves, it provides better input to the RFM, which in turn produces cleaner supervision signals.
Results and Discussion
WeCLIP was evaluated on the PASCAL VOC 2012 and MS COCO 2014 datasets. On PASCAL VOC, it achieved 77.2% mIoU, outperforming the previous state-of-the-art single-stage method by 5.0%. On the more complex MS COCO dataset, it reached 47.1% mIoU, a 4.8% improvement over prior work.
Beyond performance, the model is highly efficient, reaching peak accuracy in just 4.5 hours on a single NVIDIA RTX 3090. Furthermore, WeCLIP shows strong performance in fully supervised settings, surpassing established models like DeepLabV3 while using significantly fewer learnable parameters.
Conclusion
WeCLIP demonstrates that a frozen CLIP backbone is a powerful and efficient alternative for WSSS. While the model shows some sensitivity to text prompts and decoder depth, its ability to achieve state-of-the-art results with minimal computational resources marks a significant step toward sustainable and accessible large-scale vision modeling. Future work will explore extending this "frozen backbone" paradigm to other dense prediction tasks like object detection.
Summary updated