Introduction
While "foundation models" (FMs) like CLIP and Stable Diffusion have revolutionized image recognition and generation, simply "plugging them in" to dense tasks like image segmentation is often sub-optimal. This paper critiques current approaches, revealing that while CLIP has surprising latent localization abilities, generative models like Stable Diffusion suffer from significant feature degradation when used naively for discriminative tasks.
The Latent Localization of VLMs
The authors find that Visual-Language Models (VLMs) like CLIP possess inherent localization abilities. By analyzing internal "Value" (V) embeddings and attention maps, researchers can extract object masks without extensive retraining. This suggests that CLIP is, in many ways, a "hidden" segmenter.
However, this capability is highly sensitive to prompt engineering. The choice of text prompt (e.g., "a photo of [classname]") significantly impacts localization. Without careful tuning, the latent features can become misaligned, highlighting incorrect regions of the image.
The Generative vs. Discriminative Gap
A major contribution of this work is the critique of using generative models like Stable Diffusion for discriminative tasks. In models like ODISE, a real image is fed into the pipeline with a single step of noise to extract features. The authors demonstrate that while "cross-attention" maps are sharp during the final stages of generating an image, they become blurry and lose localization accuracy when forced to process a real noisy image.
High-quality features in diffusion models are a product of the multi-step "coarse-to-fine" denoising process. Using a "naive" one-step noise injection bypasses this process, leading to degraded semantic information.
Benchmarking and Complexity
The study compares complex diffusion-based systems like ODISE against simpler CLIP-based models like SAN. Surprisingly, SAN (using CLIP + a light decoder) often outperforms the much more complex ODISE on several open-vocabulary benchmarks:
- A-847: SAN (12.4) vs. ODISE (11.1)
- PC-459: SAN (15.7) vs. ODISE (14.5)
This suggests that the added complexity of generative pipelines does not always translate to better dense recognition, and simpler architectures leveraging CLIP's inherent localization may be more effective.
Conclusion
The paper concludes that current methods for repurposing foundation models for segmentation are relatively naive. Future research should focus on how to better exploit the multi-step denoising trajectory of diffusion models to extract higher-quality features for real-world images, rather than treating them as "black box" feature extractors.
Summary updated