SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference

Feng Wang, Jieru Mei, Alan Yuille

TL;DR

Contrastive Language-Image Pretraining (CLIP) excels at image-level classification but struggles with dense, pixel-level tasks like semantic segmentation due to spatial feature misalignment. SCLIP fixes this by introducing Correlative Self-Attention (CSA)—a simple, training-free modification to CLIP's final attention layer that reuses existing weights to produce highly localized, spatial-covariant features. This minimal change boosts zero-shot semantic segmentation performance from 14.1% to 38.2% average mIoU, outperforming existing state-of-the-art methods without any fine-tuning.

What Problem Does This Paper Address?

CLIP is fantastic at zero-shot classification because it aligns global image features with text embeddings. However, when you try to use its patch-level features directly for dense prediction tasks like semantic segmentation, performance plummets (yielding a mere 3.1% mIoU on ADE20k).

The authors discover an intriguing paradox: CLIP actually recognizes which objects are present in an image, but it completely blurs where they are. This stems from CLIP's self-attention mechanism, which learns spatial-invariant features. Because CLIP is trained to map an entire image to a single text prompt, its final layer attention maps look nearly identical regardless of which pixel or patch you query. For dense prediction tasks, we need spatial-covariant features, where local representations change dynamically according to their exact spatial positions.

Fig. 2: Final layer attention maps of vanilla CLIP with a ViT-Base/16 image encoder. We display the attention maps of four points (marked in different colors) for each example. It shows that each local visual token attends to a wide range of positions and the attention maps often share similar patterns, indicating that CLIP learns spatial- invariant visual features.

Methods

The authors propose Correlative Self-Attention (CSA) to replace the standard self-attention mechanism in the very last layer of CLIP's vision encoder. Standard self-attention uses distinct query ($W_q$) and key ($W_k$) projection matrices to aggregate global context. CSA, instead, computes attention scores based on pairwise correlations, encouraging a token to attend heavily to itself and to semantically similar regions.

Crucially, because CSA is robust to the exact projection weights, the authors don't need to train a new matrix from scratch. Instead, they reuse the pretrained $W_q$ and $W_k$ matrices from CLIP symmetrically:

$$\text{Attn} = \text{Softmax}(\mathbf{X}\mathbf{W}_q\mathbf{W}_q^T\mathbf{X}^T/\tau) + \text{Softmax}(\mathbf{X}\mathbf{W}_k\mathbf{W}_k^T\mathbf{X}^T/\tau)$$

This makes SCLIP entirely training-free and parameter-free, acting as a plug-and-play decoder head for zero-shot segmentation.

Fig. 3: An architectural comparison between the original self-attention and our cor- relative self-attention mechanism. Our method determines attention scores by pairwise correlations between the local tokens.

Datasets

The method is evaluated across eight standard semantic segmentation benchmarks:

  • PASCAL VOC 2012 (VOC21 with background, VOC20 without background)
  • PASCAL Context (Context60 with background, Context59 without background)
  • Cityscapes
  • ADE20k
  • COCO-Stuff
  • COCO-Object

No new datasets were introduced; the focus remains entirely on zero-shot transfer to existing, widely accepted benchmarks.

Key Insights and Ideas

  • CLIP's Misalignment is Architectural, Not Conceptual: CLIP already possesses the granular semantic knowledge required for segmentation, but its standard attention layer discards spatial localization in favor of global classification features.
  • Symmetric Projections Restore Locality: By forcing the attention mechanism to use query-to-query ($\mathbf{W}_q\mathbf{W}_q^T$) and key-to-key ($\mathbf{W}_k\mathbf{W}_k^T$) similarities, the diagonal elements of the attention matrix are maximized. This ensures each token primarily attends to its own position while smoothly expanding to regions with high semantic similarity.
  • Structure Over Weights: Ablation studies revealed that even using random matrices or identity projections for CSA yielded respectable segmentation results. The structural change to pairwise correlation is what restores spatial covariance, making the exact weights secondary.
Fig. 4: Comparison of attention maps. We show the attention maps of the last trans- former layer in CLIP vision encoder equipped with the original self-attention (right) and our correlative self-attention (left). Our correlative self-attention exhibits spatially covariant patterns as the attention maps are distinct to different source points and show clear boundaries of semantic objects (e.g., the chair and the cat).

Key Results

  • Massive Performance Leap: SCLIP achieves an average zero-shot mIoU of 38.2% across the eight benchmarks, a massive jump from vanilla CLIP's 14.1%.
  • Beating the State-of-the-Art: It outperforms previous open-vocabulary methods like MaskCLIP (30.3%), GroupViT (30.7%), and TCL (33.9%) by a large margin, despite requiring zero training or fine-tuning.
  • Robustness Without Post-Processing: Unlike other methods that rely heavily on expensive post-processing techniques like PAMR or DenseCRF to smooth out noisy masks, SCLIP produces clean, sharp boundaries naturally.
Fig. 1: Open-vocabulary semantic segmentation examples. We evaluate on two images from COCO [5] (the 3rd and the 5th examples) and three high-resolution images in the wild, where our SCLIP consistently generates high quality segmentation masks yet the original CLIP fails to correctly localize objects. We display the corresponding text query of each segmentation mask, where “g. retriever” and “b. collie” in the first example denote golden retriever and border collie, respectively.

Limitations and Future Work

  • Layer-Specific Application: The method is currently applied only to the last layer of the vision encoder. Exploring how spatial invariance develops across earlier layers could yield deeper insights.
  • The Supervised Gap: While SCLIP performs exceptionally well for a training-free method, fully supervised models still hold an edge (around 40% mIoU on ADE20k/COCO-Stuff vs. SCLIP's 16.1%/22.4%).
  • Broader Dense Tasks: The authors suggest that future work could involve extending this correlative attention mechanism to other dense downstream tasks like object detection or depth estimation.

Summary updated

Abstract

Recent advances in contrastive language-image pretraining (CLIP) have demonstrated strong capabilities in zero-shot classification by aligning visual representations with target text embeddings in an image level. However, in dense prediction tasks, CLIP often struggles to localize visual features within an image and fails to give accurate pixel-level predictions, which prevents it from functioning as a generalized visual foundation model. In this work, we aim to enhance CLIP's potential for semantic segmentation with minimal modifications to its pretrained models. By rethinking self-attention, we surprisingly find that CLIP can adapt to dense prediction tasks by simply introducing a novel Correlative Self-Attention (CSA) mechanism. Specifically, we replace the traditional self-attention block of CLIP vision encoder's last layer by our CSA module and reuse its pretrained projection matrices of query, key, and value, leading to a training-free adaptation approach for CLIP's zero-shot semantic segmentation. Extensive experiments show the advantage of CSA: we obtain a 38.2% average zero-shot mIoU across eight semantic segmentation benchmarks highlighted in this paper, significantly outperforming the existing SoTA's 33.9% and the vanilla CLIP's 14.1%.
BibTeX
@inproceedings{wang2024sclip,
  author = {Feng Wang and Jieru Mei and Alan Yuille},
  title = {{SCLIP: Rethinking Self-Attention for Dense Vision-Language Inference}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72664-4\_18},
}
Figure