CLIP Meets Diffusion: A Synergistic Approach to Anomaly Detection

Byeongchan Lee, John Won, Seunghyun Lee, Jinwoo Shin

TL;DR

CLIPFUSION combines the global semantic power of CLIP with the fine-grained spatial precision of Diffusion models for zero-shot and few-shot anomaly detection. By introducing an "empty mask" trick to extract intermediate features from an off-the-shelf diffusion model without retraining or iterative denoising, it achieves state-of-the-art accuracy while running 6–8× faster than window-based CLIP baselines.

What Problem Does This Paper Address?

Anomaly detection—identifying and localizing defects in images—is critical for industrial inspection and medical diagnosis. However, it is notoriously challenging because anomalies are rare, highly diverse, and context-dependent.

Recently, researchers have turned to vision-language foundation models like CLIP to build zero-shot or few-shot detectors. While CLIP excels at understanding high-level global semantics, it struggles with fine-grained local details (like tiny cracks or surface scratches). Prior attempts to fix this, such as WinCLIP, rely on sliding windows at multiple scales. This brute-force approach requires hundreds of forward passes per image, making it computationally expensive and slow.

The authors of this paper ask: Can we achieve precise, fine-grained anomaly detection without the computational nightmare of sliding windows?

Methods

The authors introduce CLIPFUSION, a framework that harmonizes a discriminative vision-language model (PatchCLIP) with a generative text-to-image model (MapDiff) without requiring any additional training.

Figure 2: Overall framework for CLIPFUSION. The CLIP-based model and the diffusion-based model process query and reference images to generate anomaly maps. The final anomaly map is obtained by fusing the outputs of the vision and language components of both models.

1. PatchCLIP (The Global View)

Instead of sliding windows, PatchCLIP processes the image in a single forward pass. It extracts patch embeddings and utilizes a specialized "VV-attention" (value-value attention) mechanism to capture local interactions between patches, providing a strong spatial prior.

2. MapDiff (The Local View)

To capture fine-grained details, the authors turn to a pretrained Stable Diffusion model. Normally, diffusion models are slow because they generate images step-by-step from random noise. CLIPFUSION bypasses this with a clever hack: the empty mask trick.

They feed a clean query image into a diffusion inpainting pipeline but provide an empty mask. This prevents any actual noise from being added, but it tricks the denoiser into processing the image at a specific timestep. They can then deterministically extract highly detailed cross-attention maps and feature maps from the intermediate decoder layers in a single pass.

3. Fusion Strategy

  • Zero-shot: The model prompts the diffusion denoiser with text like "a photo of a [object] with [state]", where [state] loops through common anomalies (crack, hole, damage). The resulting cross-attention maps are averaged and fused with CLIP’s language score maps.
  • Few-shot: When a few normal reference images are available, CLIPFUSION builds a feature memory bank from both CLIP and the diffusion decoder, calculating anomaly scores based on how much the query image deviates from the normal reference features.
Figure 3: Examples of cross-attention maps extracted from a diffusion denoiser.

Datasets

The method was evaluated on two standard industrial anomaly detection benchmarks:

  • MVTec-AD: 15 categories (objects and textures) with real-world defects like scratches, dents, and structural deformations.
  • VisA: 12 categories containing complex structures and multiple objects, presenting a higher baseline difficulty.

Key Insights and Ideas

  • Complementary Alignment: Discriminative models (CLIP) directly align text and image embeddings in a shared space, which favors global context. Generative models (Diffusion) align text and images indirectly by forcing the network to solve a pixel-level generation task, making their internal features inherently better at preserving local spatial layouts.
  • The Power of the Empty Mask: This trick turns a stochastic, iterative generative model into a fast, deterministic feature extractor perfect for discriminative downstream tasks.
  • Timestep-Block Regularization: The authors discovered that matching higher timesteps (which represent coarse noise) with lower decoder blocks (which process coarse-grained information) acts as an effective regularization strategy, suppressing local noise and highlighting broader anomaly patterns.

Key Results

  • Superior Accuracy: In zero-shot anomaly segmentation on MVTec-AD, CLIPFUSION achieved an AUROC of 90.9% and an AUPRO of 87.1%, dramatically outperforming WinCLIP (85.1% AUROC / 64.6% AUPRO).
  • Massive Speedup: By eliminating multi-scale sliding windows, CLIPFUSION cuts inference latency significantly. In the zero-shot setting, it runs 6 to 8 times faster than WinCLIP (783ms vs 4782ms on an NVIDIA A100).
  • Few-Shot Dominance: CLIPFUSION consistently outperformed both vision-only baselines (PatchCore, PaDiM) and training-based vision-language baselines (PromptAD) across 1, 2, and 4-shot settings, demonstrating excellent data scalability and lower performance variance.
Figure 1: Qualitative results of our model CLIPFUSION. It captures fine-grained anomalies more effectively than WinCLIP.

Limitations and Future Work

Despite its strong performance, the authors note two primary failure modes:

  1. Attention Saturation: On dense, highly repetitive patterns (like tightly woven wire meshes), the diffusion model tends to over-activate across the entire pattern, occasionally missing the true localized defect.
  2. Logical Anomalies: Because CLIPFUSION relies heavily on texture and local feature deviations, it struggles with "logical" anomalies—such as a circuit board that is physically perfect but missing a vital transistor. The model lacks the high-level structural logic to realize an object's absence is the defect.

Future work includes improving the model's contextual and logical reasoning to address missing-element defects.

Summary updated

Abstract

Anomaly detection is a complex problem due to the ambiguity in defining anomalies, the diversity of anomaly types (e.g., local and global defect), and the scarcity of training data. As such, it necessitates a comprehensive model capable of capturing both low-level and high-level features, even with limited data. To address this, we propose CLIPFUSION, a method that leverages both discriminative and generative foundation models. Specifically, the CLIP-based discriminative model excels at capturing global features, while the diffusion-based generative model effectively captures local details, creating a synergistic and complementary approach. Notably, we introduce a methodology for utilizing cross-attention maps and feature maps extracted from diffusion models specifically for anomaly detection. Experimental results on benchmark datasets (MVTec-AD, VisA) demonstrate that CLIPFUSION consistently outperforms baseline methods, achieving outstanding performance in both anomaly segmentation and classification. We believe that our method underscores the effectiveness of multi-modal and multi-model fusion in tackling the multifaceted challenges of anomaly detection, providing a scalable solution for real-world applications.
BibTeX
@article{lee2025clip,
  author = {Byeongchan Lee and John Won and Seunghyun Lee and Jinwoo Shin},
  title = {{CLIP Meets Diffusion: A Synergistic Approach to Anomaly Detection}},
  journal = {Accepted by TMLR},
  year = {2025},
  url = {https://openreview.net/forum?id=WpFzZNuQmg},
}
Figure