Abstract
The core idea of visual anomaly detection is to learn the normality from normal images, but previous works have been developed specifically for certain tasks, leading to fragmentation among various tasks: defect detection, semantic anomaly detection, multi-class anomaly detection, and anomaly clustering. This one-task-one-model approach is resource-intensive and incurs high maintenance costs as the number of tasks increases. This paper presents SelFormaly, a universal and powerful anomaly detection framework. We emphasize the necessity of our off-the-shelf approach by pointing out a suboptimal issue with fluctuating performance in previous online encoder-based methods. In addition, we question the effectiveness of using ConvNets as previously employed in the literature and confirm that self-supervised ViTs are suitable for unified anomaly detection. We introduce back-patch masking and discover the new role of top k-ratio feature matching to achieve unified and powerful anomaly detection. Back-patch masking eliminates irrelevant regions that possibly hinder target-centric detection with representations of the scene layout. The top k-ratio feature matching unifies various anomaly levels and tasks. Finally, SelFormaly achieves state-of-the-art results across various datasets for all the aforementioned tasks.
Introduction
Visual Anomaly Detection (AD) has traditionally been treated as a collection of isolated problems. Industrial defect detection (e.g., identifying cracks in metal) and semantic anomaly detection (e.g., distinguishing a cat from a dog) have relied on distinct architectures and training paradigms. This fragmentation leads to a "one-task-one-model" bottleneck, which is increasingly unsustainable as the diversity of applications grows.
SelFormaly addresses this by proposing a unified, task-agnostic framework. Unlike previous methods that rely on online learning and proxy tasks—which often suffer from performance fluctuations—SelFormaly utilizes an off-the-shelf approach leveraging self-supervised Vision Transformers (ViTs).
Methodology
The SelFormaly framework is built upon two primary innovations designed to handle the noise and scale variations inherent in unified AD.
Back-Patch Masking (BPM)
A significant challenge in anomaly detection is the presence of irrelevant background information. If a model incorporates background features into its "normality" memory bank, environmental changes can trigger false positives. SelFormaly employs Back-Patch Masking, which utilizes the internal self-attention maps of a pre-trained ViT (such as DINO or iBOT) to identify the primary object of interest. By automatically masking out background patches, the model focuses exclusively on target-centric features, improving detection accuracy and reducing memory overhead.
Top $k$-ratio Feature Matching
To bridge the gap between local defects and global semantic anomalies, SelFormaly treats AD as a Multiple Instance Learning (MIL) problem. Traditional methods often rely on either the single most anomalous patch (which is prone to noise) or global averaging (which dilutes small defects). SelFormaly instead calculates the anomaly score based on the top $k\%$ of anomalous patches. This $k$-ratio acts as a universal hyperparameter; the authors found that a value of approximately 5% effectively unifies detection across tiny industrial scratches and large-scale semantic shifts.
Experimental Evaluation
The framework was evaluated across a diverse set of benchmarks, including MVTecAD, BTAD, CIFAR, and the high-resolution Species-60 dataset.
Key Performance Metrics
- Industrial Benchmarks: On MVTecAD, SelFormaly achieved a state-of-the-art image-level AUROC of 99.32.
- Semantic AD: On the Species-60 dataset, it reached 94.4 AUROC, significantly outperforming specialized competitors.
- Multi-class and Clustering: The model maintains high performance even when learning multiple normal classes simultaneously, a scenario where previous models typically struggle. In anomaly clustering tasks, it improved the F1 score by over 10% compared to existing baselines.
Conclusion
SelFormaly demonstrates that a unified approach to anomaly detection is not only possible but superior to task-specific models. By leveraging self-supervised ViT features and robust statistical matching techniques like top $k$-ratio, the framework provides a scalable, low-maintenance solution for a wide array of visual inspection and classification tasks. Future work will explore extending this task-agnostic logic to 3D data and multimodal inputs.
Summary updated