Breaking the Bias: Recalibrating the Attention of Industrial Anomaly Detection

Xin Chen, Liujuan Cao, Shengchuan Zhang, Xiewu Zheng, Yan ZhangarXiv 2024 · DOI · arXiv

Abstract

Industrial Anomaly Detection (IAD) typically relies on unsupervised learning from normal samples, but these models often suffer from "attention bias," focusing on natural variations while ignoring invariant regions where subtle defects may occur. This paper introduces RAAD (Recalibrating Attention of Industrial Anomaly Detection), a framework that addresses this bias through Hierarchical Quantization Scoring (HQS) and targeted fine-tuning. By intentionally reducing precision in specific layers, RAAD regularizes the model to ignore noise and focus on critical features. Our approach achieves state-of-the-art results on MVTec AD, MVTec LOCO, and VisA benchmarks, maintaining high inference speeds suitable for real-time industrial applications.

Introduction

In industrial inspection, defects are rare and unpredictable, necessitating models that can learn from "normal" data alone. However, unsupervised models often develop a subtle but pervasive problem: they overfit to irrelevant variations in normal data—such as lighting shifts or minor positional changes—while ignoring the stable regions where defects are most likely to be noticed. This "attention bias" can lead to missed detections of subtle cracks or missing components.

The Problem of Attention Bias

In a perfect scenario, an inspection model would monitor every part of a product with equal sensitivity. In reality, models tend to focus on regions that vary naturally. If a defect appears in a "boring," invariant region, the model may fail to flag it because it wasn't paying attention to that area during training. To solve this, we must decompose the model's biased attention and recalibrate it to be sensitive to all defect-prone areas.

Visualizing attention bias: vanilla models often focus on natural variations (Normal map) rather than the entire object, potentially missing anomalies in stable regions.

The RAAD Framework

The RAAD framework operates in three primary stages to break this bias and improve detection accuracy:

  1. Initialization: A standard teacher-student architecture is employed. A pre-trained "teacher" network provides a reference, and a "student" network attempts to mimic its feature extraction. Discrepancies between the two on new images indicate potential anomalies.
  2. Attention Map Decomposition via Hierarchical Quantization Scoring (HQS): Instead of high-precision computation, RAAD intentionally reduces the bit-width of the model's layers. This quantization acts as a regularizer, forcing the model to ignore low-level noise and focus on the most critical semantic features. HQS dynamically allocates more bits to deeper layers (which provide sharp focus) and fewer bits to shallower, noisier layers.
  3. Attention Map Recalibration: Following quantization, the model undergoes fine-tuning to redistribute attention toward regions susceptible to defects. This stage ensures that the model's sensitivity is balanced across the entire object.
The RAAD pipeline: (a) demonstrates the evolution of anomaly maps from vanilla to quantized and fine-tuned states; (b) illustrates the refinement of feature maps across successive convolutional layers.

To capture both local defects (like scratches) and logical defects (like missing components), RAAD integrates the teacher-student output with a global autoencoder, providing a comprehensive view of the sample.

Experimental Results

RAAD was validated across 32 datasets in three major benchmarks, consistently outperforming heavier models:

  • MVTec AD: Achieved a 98.9% AU-ROC for detection and 92.92% PRO for segmentation.
  • MVTec LOCO: RAAD excelled on this challenging dataset focused on logical constraints, reaching a mean AU-ROC of 89.75%, a significant improvement over previous state-of-the-art models like EfficientAD-S (84.09%).
  • Efficiency: The model maintains high performance without sacrificing speed, achieving an inference rate of 113 FPS on a single NVIDIA RTX 3090ti.

Conclusion and Future Work

RAAD demonstrates that quantization can be used as a powerful tool for feature regularization rather than just a method for model compression. By breaking the attention bias inherent in unsupervised IAD, RAAD sets a new standard for accuracy and efficiency. Future research will focus on reducing dependency on pre-trained teachers and exploring more complex, dynamic backgrounds in factory settings.

Summary updated

Abstract

Due to the scarcity and unpredictable nature of defect samples, industrial anomaly detection (IAD) predominantly employs unsupervised learning. However, all unsupervised IAD methods face a common challenge: the inherent bias in normal samples, which causes models to focus on variable regions while overlooking potential defects in invariant areas. To effectively overcome this, it is essential to decompose and recalibrate attention, guiding the model to suppress irrelevant variations and concentrate on subtle, defect-susceptible areas. In this paper, we propose Recalibrating Attention of Industrial Anomaly Detection (RAAD), a framework that systematically decomposes and recalibrates attention maps. RAAD employs a two-stage process: first, it reduces attention bias through quantization, and second, it fine-tunes defect-prone regions for improved sensitivity. Central to this framework is Hierarchical Quantization Scoring (HQS), which dynamically allocates bit-widths across layers based on their anomaly detection contributions. HQS dynamically adjusts bit-widths based on the hierarchical nature of attention maps, compressing lower layers that produce coarse and noisy attention while preserving deeper layers with sharper, defect-focused attention. This approach optimizes both computational efficiency and the model' s sensitivity to anomalies. We validate the effectiveness of RAAD on 32 datasets using a single 3090ti. Experiments demonstrate that RAAD, balances the complexity and expressive power of the model, enhancing its anomaly detection capability.
BibTeX
@misc{chen2024breaking,
  author = {Xin Chen and Liujuan Cao and Shengchuan Zhang and Xiewu Zheng and Yan Zhang},
  title = {{Breaking the Bias: Recalibrating the Attention of Industrial Anomaly Detection}},
  year = {2024},
  eprint = {2412.08189},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.08189},
}
Figure