Domain Adaptation of Attention Heads for Zero-shot Anomaly Detection

Kiyoon Jeong, Jaehyuk Heo, Junyeong Son, Pilsung KangarXiv 2025 · DOI · arXiv

Introduction

In fields such as industrial quality control and medical imaging, anomaly detection is critical for identifying defects or pathologies. Traditional AI models require extensive datasets of "normal" images to learn deviations. However, in scenarios involving new product lines or rare diseases, such data is often scarce or restricted. Zero-Shot Anomaly Detection (ZSAD) addresses this "cold start" problem by leveraging massive pre-trained vision-language models like CLIP to identify anomalies without target-specific training.

Despite its power, CLIP faces a significant "domain gap" when transitioning from natural internet images to specialized industrial or medical scans. Existing adaptation methods often rely on heavy adapter modules or prompt engineering that fails to fully bridge this gap efficiently. This paper introduces HeadCLIP, a framework that adapts CLIP's attention mechanism with minimal overhead to achieve state-of-the-art performance in specialized domains.

Methodology: Learnable Head Weights

HeadCLIP's core innovation is the introduction of Learnable Head Weights (LHW) within the Vision Transformer (ViT) architecture. Rather than retraining the entire network, HeadCLIP assigns a single learnable scalar weight to each attention head in the local feature pathway. This allows the model to selectively amplify heads that are sensitive to domain-specific features (e.g., textures or spatial structures) while suppressing those tuned to irrelevant internet-specific noise.

The HeadCLIP architecture featuring the Multi-Head Consistent Self-Attention (MHCSA) layers and the Joint Anomaly Score (JAS) calculation.

In addition to LHW, the framework utilizes:

  • Class-Agnostic Prompts: Learnable text prompts (e.g., "[Tokens] object" vs. "[Tokens] damaged object") prevent overfitting to specific categories, enhancing generalization.
  • Joint Anomaly Score (JAS): This mechanism fuses global image-level scores with local scores derived from the top-k% most suspicious pixels, ensuring that subtle, localized defects are not overlooked by global embeddings.

Experimental Results and Efficiency

The researchers evaluated HeadCLIP across 17 public datasets spanning industrial (e.g., MVTec AD, VisA) and medical (e.g., ISIC, BrainMRI) domains. Notably, the model was trained on auxiliary industrial data and tested completely zero-shot on medical datasets to demonstrate its cross-domain robustness.

Performance Gains

HeadCLIP consistently outperformed existing state-of-the-art methods:

  • Industrial Domain: Achieved up to a 4.9 percentage point (%p) improvement in pixel-level mean Anomaly Detection (mAD).
  • Medical Domain: Despite being trained only on industrial data, it achieved a +3.2%p improvement in both pixel-level and image-level mAD.

Computational Efficiency

A primary advantage of HeadCLIP is its extreme parameter efficiency. By focusing on attention head reweighting, the model adds only 304 visual parameters, compared to the millions required by competing adapter-based methods.

Comparison of HeadCLIP against other ZSAD models, highlighting the minimal visual parameter count required for adaptation.

Conclusion and Future Work

HeadCLIP demonstrates that effective domain adaptation does not require massive architectural changes. By reweighting existing attention heads, the model bridges the gap between general-purpose pre-training and specialized application.

However, limitations remain, such as a "single-object bias" inherited from training on centered datasets like MVTec AD, which can hinder performance in cluttered scenes. Future research will focus on training with more complex, multi-object environments and developing dynamic head weights that can adjust during inference to further enhance zero-shot capabilities.

Summary updated

BibTeX
@misc{jeong2025domain,
  author = {Kiyoon Jeong and Jaehyuk Heo and Junyeong Son and Pilsung Kang},
  title = {{Domain Adaptation of Attention Heads for Zero-shot Anomaly Detection}},
  year = {2025},
  eprint = {2505.22259},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.22259},
}
Figure