SelFormaly: Towards Task-Agnostic Unified Anomaly Detection

Yujin Lee, Harin Lim, Hyunsoo Yoon

Abstract

The core idea of visual anomaly detection is to learn the normality from normal images, but previous works have been developed specifically for certain tasks, leading to fragmentation among various tasks: defect detection, semantic anomaly detection, multi-class anomaly detection, and anomaly clustering. This one-task-one-model approach is resource-intensive and incurs high maintenance costs as the number of tasks increases. This paper presents SelFormaly, a universal and powerful anomaly detection framework. We emphasize the necessity of our off-the-shelf approach by pointing out a suboptimal issue with fluctuating performance in previous online encoder-based methods. In addition, we question the effectiveness of using ConvNets as previously employed in the literature and confirm that self-supervised ViTs are suitable for unified anomaly detection. We introduce back-patch masking and discover the new role of top k-ratio feature matching to achieve unified and powerful anomaly detection. Back-patch masking eliminates irrelevant regions that possibly hinder target-centric detection with representations of the scene layout. The top k-ratio feature matching unifies various anomaly levels and tasks. Finally, SelFormaly achieves state-of-the-art results across various datasets for all the aforementioned tasks.

Introduction

Visual Anomaly Detection (AD) has traditionally been treated as a collection of isolated problems. Industrial defect detection (e.g., identifying cracks in metal) and semantic anomaly detection (e.g., distinguishing a cat from a dog) have relied on distinct architectures and training paradigms. This fragmentation leads to a "one-task-one-model" bottleneck, which is increasingly unsustainable as the diversity of applications grows.

Standard defect detection tasks across various industrial categories like metal components and wood textures.

SelFormaly addresses this by proposing a unified, task-agnostic framework. Unlike previous methods that rely on online learning and proxy tasks—which often suffer from performance fluctuations—SelFormaly utilizes an off-the-shelf approach leveraging self-supervised Vision Transformers (ViTs).

Methodology

The SelFormaly framework is built upon two primary innovations designed to handle the noise and scale variations inherent in unified AD.

Back-Patch Masking (BPM)

A significant challenge in anomaly detection is the presence of irrelevant background information. If a model incorporates background features into its "normality" memory bank, environmental changes can trigger false positives. SelFormaly employs Back-Patch Masking, which utilizes the internal self-attention maps of a pre-trained ViT (such as DINO or iBOT) to identify the primary object of interest. By automatically masking out background patches, the model focuses exclusively on target-centric features, improving detection accuracy and reducing memory overhead.

Top $k$-ratio Feature Matching

To bridge the gap between local defects and global semantic anomalies, SelFormaly treats AD as a Multiple Instance Learning (MIL) problem. Traditional methods often rely on either the single most anomalous patch (which is prone to noise) or global averaging (which dilutes small defects). SelFormaly instead calculates the anomaly score based on the top $k\%$ of anomalous patches. This $k$-ratio acts as a universal hyperparameter; the authors found that a value of approximately 5% effectively unifies detection across tiny industrial scratches and large-scale semantic shifts.

Experimental Evaluation

The framework was evaluated across a diverse set of benchmarks, including MVTecAD, BTAD, CIFAR, and the high-resolution Species-60 dataset.

Multi-class anomaly detection demonstrating the model's ability to distinguish normal from anomalous samples across diverse domains.

Key Performance Metrics

  • Industrial Benchmarks: On MVTecAD, SelFormaly achieved a state-of-the-art image-level AUROC of 99.32.
  • Semantic AD: On the Species-60 dataset, it reached 94.4 AUROC, significantly outperforming specialized competitors.
  • Multi-class and Clustering: The model maintains high performance even when learning multiple normal classes simultaneously, a scenario where previous models typically struggle. In anomaly clustering tasks, it improved the F1 score by over 10% compared to existing baselines.

Conclusion

SelFormaly demonstrates that a unified approach to anomaly detection is not only possible but superior to task-specific models. By leveraging self-supervised ViT features and robust statistical matching techniques like top $k$-ratio, the framework provides a scalable, low-maintenance solution for a wide array of visual inspection and classification tasks. Future work will explore extending this task-agnostic logic to 3D data and multimodal inputs.

Summary updated

Abstract

The core idea of visual anomaly detection is to learn the normality from normal images, but previous works have been developed specifically for certain tasks, leading to fragmentation among various tasks: defect detection, semantic anomaly detection, multi-class anomaly detection, and anomaly clustering. This one-task-one-model approach is resource-intensive and incurs high maintenance costs as the number of tasks increases. This paper presents SelFormaly, a universal and powerful anomaly detection framework. We emphasize the necessity of our off-the-shelf approach by pointing out a suboptimal issue with fluctuating performance in previous online encoder-based methods. In addition, we question the effectiveness of using ConvNets as previously employed in the literature and confirm that self-supervised ViTs are suitable for unified anomaly detection. We introduce back-patch masking and discover the new role of top k-ratio feature matching to achieve unified and powerful anomaly detection. Back-patch masking eliminates irrelevant regions that possibly hinder target-centric detection with representations of the scene layout. The top k-ratio feature matching unifies various anomaly levels and tasks. Finally, SelFormaly achieves state-of-the-art results across various datasets for all the aforementioned tasks.
BibTeX
@article{lee2026selformaly,
  author = {Yujin Lee and Harin Lim and Hyunsoo Yoon},
  title = {{SelFormaly: Towards Task-Agnostic Unified Anomaly Detection}},
  journal = {Pattern Recognition},
  year = {2026},
  doi = {10.1016/j.patcog.2025.111820},
}

Loading related content...

Figure