A Critical Look at the Current Usage of Foundation Model for Dense Recognition Task

Shiqi Yang, Atsushi Hashimoto, Yoshitaka UshikuarXiv 2023 · DOI · arXiv

Introduction

While "foundation models" (FMs) like CLIP and Stable Diffusion have revolutionized image recognition and generation, simply "plugging them in" to dense tasks like image segmentation is often sub-optimal. This paper critiques current approaches, revealing that while CLIP has surprising latent localization abilities, generative models like Stable Diffusion suffer from significant feature degradation when used naively for discriminative tasks.

The Latent Localization of VLMs

The authors find that Visual-Language Models (VLMs) like CLIP possess inherent localization abilities. By analyzing internal "Value" (V) embeddings and attention maps, researchers can extract object masks without extensive retraining. This suggests that CLIP is, in many ways, a "hidden" segmenter.

Successful localization of the 'car' token in a CLIP-based feature map, showing the model's latent ability to identify object boundaries.

However, this capability is highly sensitive to prompt engineering. The choice of text prompt (e.g., "a photo of [classname]") significantly impacts localization. Without careful tuning, the latent features can become misaligned, highlighting incorrect regions of the image.

A failure case where the 'trees' token incorrectly highlights the car region, demonstrating the sensitivity of latent features to specific prompts.

The Generative vs. Discriminative Gap

A major contribution of this work is the critique of using generative models like Stable Diffusion for discriminative tasks. In models like ODISE, a real image is fed into the pipeline with a single step of noise to extract features. The authors demonstrate that while "cross-attention" maps are sharp during the final stages of generating an image, they become blurry and lose localization accuracy when forced to process a real noisy image.

High-quality features in diffusion models are a product of the multi-step "coarse-to-fine" denoising process. Using a "naive" one-step noise injection bypasses this process, leading to degraded semantic information.

Benchmarking and Complexity

The study compares complex diffusion-based systems like ODISE against simpler CLIP-based models like SAN. Surprisingly, SAN (using CLIP + a light decoder) often outperforms the much more complex ODISE on several open-vocabulary benchmarks:

  • A-847: SAN (12.4) vs. ODISE (11.1)
  • PC-459: SAN (15.7) vs. ODISE (14.5)

This suggests that the added complexity of generative pipelines does not always translate to better dense recognition, and simpler architectures leveraging CLIP's inherent localization may be more effective.

Conclusion

The paper concludes that current methods for repurposing foundation models for segmentation are relatively naive. Future research should focus on how to better exploit the multi-step denoising trajectory of diffusion models to extract higher-quality features for real-world images, rather than treating them as "black box" feature extractors.

Summary updated

Abstract

In recent years large model trained on huge amount of cross-modality data, which is usually be termed as foundation model, achieves conspicuous accomplishment in many fields, such as image recognition and generation. Though achieving great success in their original application case, it is still unclear whether those foundation models can be applied to other different downstream tasks. In this paper, we conduct a short survey on the current methods for discriminative dense recognition tasks, which are built on the pretrained foundation model. And we also provide some preliminary experimental analysis of an existing open-vocabulary segmentation method based on Stable Diffusion, which indicates the current way of deploying diffusion model for segmentation is not optimal. This aims to provide insights for future research on adopting foundation model for downstream task.
BibTeX
@misc{yang2023critical,
  author = {Shiqi Yang and Atsushi Hashimoto and Yoshitaka Ushiku},
  title = {{A Critical Look at the Current Usage of Foundation Model for Dense Recognition Task}},
  year = {2023},
  eprint = {2307.02862},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.02862},
}
Figure