TL;DR
CLIPFUSION combines the global semantic power of CLIP with the fine-grained spatial precision of Diffusion models for zero-shot and few-shot anomaly detection. By introducing an "empty mask" trick to extract intermediate features from an off-the-shelf diffusion model without retraining or iterative denoising, it achieves state-of-the-art accuracy while running 6–8× faster than window-based CLIP baselines.
What Problem Does This Paper Address?
Anomaly detection—identifying and localizing defects in images—is critical for industrial inspection and medical diagnosis. However, it is notoriously challenging because anomalies are rare, highly diverse, and context-dependent.
Recently, researchers have turned to vision-language foundation models like CLIP to build zero-shot or few-shot detectors. While CLIP excels at understanding high-level global semantics, it struggles with fine-grained local details (like tiny cracks or surface scratches). Prior attempts to fix this, such as WinCLIP, rely on sliding windows at multiple scales. This brute-force approach requires hundreds of forward passes per image, making it computationally expensive and slow.
The authors of this paper ask: Can we achieve precise, fine-grained anomaly detection without the computational nightmare of sliding windows?
Methods
The authors introduce CLIPFUSION, a framework that harmonizes a discriminative vision-language model (PatchCLIP) with a generative text-to-image model (MapDiff) without requiring any additional training.
1. PatchCLIP (The Global View)
Instead of sliding windows, PatchCLIP processes the image in a single forward pass. It extracts patch embeddings and utilizes a specialized "VV-attention" (value-value attention) mechanism to capture local interactions between patches, providing a strong spatial prior.
2. MapDiff (The Local View)
To capture fine-grained details, the authors turn to a pretrained Stable Diffusion model. Normally, diffusion models are slow because they generate images step-by-step from random noise. CLIPFUSION bypasses this with a clever hack: the empty mask trick.
They feed a clean query image into a diffusion inpainting pipeline but provide an empty mask. This prevents any actual noise from being added, but it tricks the denoiser into processing the image at a specific timestep. They can then deterministically extract highly detailed cross-attention maps and feature maps from the intermediate decoder layers in a single pass.
3. Fusion Strategy
- Zero-shot: The model prompts the diffusion denoiser with text like "a photo of a [object] with [state]", where
[state]loops through common anomalies (crack, hole, damage). The resulting cross-attention maps are averaged and fused with CLIP’s language score maps. - Few-shot: When a few normal reference images are available, CLIPFUSION builds a feature memory bank from both CLIP and the diffusion decoder, calculating anomaly scores based on how much the query image deviates from the normal reference features.
Datasets
The method was evaluated on two standard industrial anomaly detection benchmarks:
- MVTec-AD: 15 categories (objects and textures) with real-world defects like scratches, dents, and structural deformations.
- VisA: 12 categories containing complex structures and multiple objects, presenting a higher baseline difficulty.
Key Insights and Ideas
- Complementary Alignment: Discriminative models (CLIP) directly align text and image embeddings in a shared space, which favors global context. Generative models (Diffusion) align text and images indirectly by forcing the network to solve a pixel-level generation task, making their internal features inherently better at preserving local spatial layouts.
- The Power of the Empty Mask: This trick turns a stochastic, iterative generative model into a fast, deterministic feature extractor perfect for discriminative downstream tasks.
- Timestep-Block Regularization: The authors discovered that matching higher timesteps (which represent coarse noise) with lower decoder blocks (which process coarse-grained information) acts as an effective regularization strategy, suppressing local noise and highlighting broader anomaly patterns.
Key Results
- Superior Accuracy: In zero-shot anomaly segmentation on MVTec-AD, CLIPFUSION achieved an AUROC of 90.9% and an AUPRO of 87.1%, dramatically outperforming WinCLIP (85.1% AUROC / 64.6% AUPRO).
- Massive Speedup: By eliminating multi-scale sliding windows, CLIPFUSION cuts inference latency significantly. In the zero-shot setting, it runs 6 to 8 times faster than WinCLIP (783ms vs 4782ms on an NVIDIA A100).
- Few-Shot Dominance: CLIPFUSION consistently outperformed both vision-only baselines (PatchCore, PaDiM) and training-based vision-language baselines (PromptAD) across 1, 2, and 4-shot settings, demonstrating excellent data scalability and lower performance variance.
Limitations and Future Work
Despite its strong performance, the authors note two primary failure modes:
- Attention Saturation: On dense, highly repetitive patterns (like tightly woven wire meshes), the diffusion model tends to over-activate across the entire pattern, occasionally missing the true localized defect.
- Logical Anomalies: Because CLIPFUSION relies heavily on texture and local feature deviations, it struggles with "logical" anomalies—such as a circuit board that is physically perfect but missing a vital transistor. The model lacks the high-level structural logic to realize an object's absence is the defect.
Future work includes improving the model's contextual and logical reasoning to address missing-element defects.
Summary updated