Repurposing CLIP to Localize at Pixel Level

Jiaxiang Fang, Shiqiang Ma, Jing Wang, Siyu Chen, Fei Guo, Shengfeng HearXiv 2026 · arXiv

TL;DR

CLIPix is a novel framework that repurposes the image-level classification power of CLIP for precise, pixel-level open-set semantic segmentation. By tracing back CLIP's classification decisions via backpropagation and applying a noise-resistant correction strategy, CLIPix achieves state-of-the-art segmentation performance while remaining highly computationally efficient.

What Problem Does This Paper Address?

Open-set semantic segmentation—the ability to identify and outline arbitrary object categories in an image, even those unseen during training—is a holy grail in computer vision. While foundation models like the Segment Anything Model (SAM) have made massive strides, they often require manual prompts or suffer from high false-positive rates when automated.

Vision-Language Models (VLMs) like CLIP offer a promising alternative because they inherently map text categories to visual concepts. However, CLIP was trained using contrastive learning on global image-text pairs. When researchers try to force CLIP to perform pixel-level localization (usually by computing an affinity matrix between text and image patches), the results are notoriously noisy and incomplete. CLIP suffers from "global feature bias" and "local bias"—for instance, it might activate strongly on a horse's head to make a classification decision, but ignore the rest of the horse's body, leading to poor segmentation masks.

Methods

The authors introduce CLIPix, a framework designed to extract clean, full-body pixel localizations directly from CLIP without losing its open-set generalization capabilities. The pipeline consists of three main stages:

  1. Initial Location via Backpropagation: Instead of relying on a coarse text-image affinity matrix, CLIPix traces back CLIP’s actual classification process. By calculating the gradients of the final classification score with respect to the feature maps of key layers, it extracts a highly specific response activation map ($L_1$).
  2. Noise-Resistant Correction (NRC): Because CLIP is sensitive to global context, $L_1$ can still contain background noise. The NRC module uses $L_1$ to re-weight the initial image patch embeddings and constructs a "pseudo-target prototype" to update CLIP's text token. Passing this refined input back through the visual encoder yields a much cleaner, denoised localization map ($L_2$).
  3. Localization Embedding (LE): To fix the local bias problem (e.g., only segmenting the head of an animal), the LE strategy uses a skip-connection-inspired architecture. It aggregates the local activation cues ($L_1, L_2$) with high-level features from a lightweight CNN encoder, expanding the model's focus from local features to full-body object awareness.

The entire architecture is illustrated below:

Fig. 2. The overview of our CLIPix framework (a). CLIPix repurposes CLIP for accurate pixel-level localization while preserving its natural generalization, unlocking its segmentation capability. From CLIP’s visual-language logits, we derive perception regions in key layers as initial localization hints (b). Our Noise-Resistant Correction strategy denoises these hints through refinement (c). The cleaned localization cues are integrated via our Localization Embedding strategy, enhancing image features with detailed object awareness and raising segmentation performance (d).

Datasets

The authors evaluated CLIPix on several standard benchmarks:

  • PASCAL-$5^i$ & COCO-$20^i$: Standard datasets for cross-validation in binary open-set semantic segmentation, split into disjoint folds of seen and unseen classes.
  • PASCAL VOC 2012, PASCAL Context (PC59), Cityscapes, and ADE20K: Used to test the model's capability when expanded to multi-class open-vocabulary segmentation tasks.

Key Insights and Ideas

  • Classification Decisions over Feature Alignment: A key "aha moment" is that tracing CLIP's mature classification decisions via backpropagation yields far more accurate localization cues than forcing a direct pixel-to-text feature alignment.
  • Overcoming the "Diagnostic Part" Trap: Classification models are lazy; they only need to find a single diagnostic feature (like a wheel for a car) to output the correct label. CLIPix successfully bridges the gap between classification and dense prediction by explicitly forcing the network to expand its attention from these diagnostic parts to the entire object body.
  • The Instrumental Object Deficit: The authors noticed a fascinating performance gap: CLIPix excels at segmenting natural objects (animals, plants) but struggles with human-made "instrumental" objects (forks, spoons, hair dryers). This points to an underlying data distribution imbalance in CLIP's original pre-training dataset.

Key Results

CLIPix achieves massive performance gains over existing zero-shot methods, even outperforming some complex few-shot models that require support images:

  • Segmentation Accuracy: On PASCAL-$5^i$, CLIPix achieved a mean Intersection over Union (mIoU) of 80.7%, outperforming previous zero-shot methods by up to 21.3%. On the challenging COCO-$20^i$ dataset, it achieved 61.8% mIoU, a 26.5% leap over prior upper bounds.
  • Efficiency: Thanks to an incredibly lightweight decoder design (only a few convolution and normalization layers), CLIPix runs at 24 FPS with a ResNet backbone and up to 50 FPS when paired with a lightweight MobileNetV2 backbone, making it highly viable for edge deployment.

Limitations and Future Work

Despite its strengths, CLIPix is still bound by CLIP's pre-existing image-level alignment, meaning it can struggle in highly cluttered or complex scenes where multiple objects overlap. Additionally, its performance on human-made tools and utensils remains a weak point.

For future work, the authors plan to explore even lighter backbones (like CLIP ViT-S) and optimize the attention mechanisms in CLIP's shallow layers to push frame rates past 90 FPS, making the framework ideal for real-time AR/VR applications.

Summary updated

Abstract

Large-scale Vision-Language Models like CLIP have demonstrated impressive open-set localization capabilities at the image level. However, adapting this capability to pixel-level dense prediction poses challenges due to global feature biases. In this paper, we introduce CLIPix, a simple yet effective framework that repurposes CLIP to perform pixel-level localization. By tracing back CLIP's classification process, CLIPix identifies object-specific attentive regions and repurposes them as pixel-level localization cues. To address noise introduced by global biases, we propose a Noise-Resistant Correction strategy, refining these cues for more precise segmentation. Additionally, we introduce a Localization Embedding strategy to integrate both localization and enriched detail information, enabling accurate, high-resolution segmentation. Our approach preserves CLIP's generalization strength and unlocks its potential for segmenting arbitrary objects. Extensive experiments on the PASCAL and COCO datasets demonstrate that CLIPix achieves state-of-the-art performance, underscoring its effectiveness.
BibTeX
@misc{fang2026repurposing,
  author = {Jiaxiang Fang and Shiqiang Ma and Jing Wang and Siyu Chen and Fei Guo and Shengfeng He},
  title = {{Repurposing CLIP to Localize at Pixel Level}},
  year = {2026},
  eprint = {2607.05253},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2607.05253},
}
Figure