UniFormaly: Towards task-agnostic unified framework for visual anomaly detection

Yujin Lee, Harin Lim, Seoyoon Jang, Hyunsoo Yoon

TL;DR

This paper introduces UniFormaly, a unified framework for visual anomaly detection that tackles multiple tasks (defect, semantic, multi-class anomaly detection, and anomaly clustering) with a single model. By leveraging self-supervised Vision Transformers and introducing novel techniques like Back Patch Masking and top k-ratio feature matching, UniFormaly achieves state-of-the-art performance across diverse tasks and datasets, offering a more efficient and robust solution than task-specific approaches.

What Problem Does This Paper Address?

Visual anomaly detection, a critical task in many industries for identifying deviations from normal patterns, is currently fragmented. Existing methods often require a separate model for each specific task (e.g., defect detection, semantic anomaly detection, multi-class anomaly detection, anomaly clustering). This "one-task-one-model" paradigm is resource-intensive, demanding significant computational power and maintenance effort as the number of tasks grows. The authors argue that this fragmentation hinders the development of a truly universal and efficient anomaly detection system. They aim to bridge this gap by proposing a single, unified framework that can handle a wide spectrum of anomaly detection tasks without task-specific design or extensive retraining.

Methods

UniFormaly is built upon self-supervised Vision Transformers (ViTs) as a foundation, leveraging their powerful pre-trained representations. The core innovations are:

  • Back Patch Masking (BPM): This technique uses self-attention maps from ViTs to identify and mask out irrelevant background regions. This reduces memory consumption, especially for large datasets, and helps focus the model on the target (normal or anomalous) regions in a task-agnostic manner. BPM is applied after smoothing the attention map to avoid over-reliance on raw attention.
  • Top k-ratio Feature Matching: This method unifies anomaly scoring across different tasks and anomaly levels. It involves extracting patch-level features, computing their distances to a memory bank of normal features, and then selecting the top k% of these distances. This approach is inspired by multiple instance learning and allows for a more robust anomaly score that considers multiple instances, rather than relying on a single highest score. This also enables anomaly clustering by grouping similar top-k features.

Datasets

The paper evaluates UniFormaly on a wide range of datasets across different anomaly detection tasks:

  • Defect Detection: MVTecAD (15 sub-datasets), Magnetic Tile Defects (MTD), BeanTech Anomaly Detection (BTAD), Casting manufacturing Product Defect (CPD).
  • Semantic Anomaly Detection: CIFAR-10, CIFAR-100, ImageNet-30, Species-60.
  • Anomaly Clustering: MVTecAD.
  • Low-shot Scenarios: MVTecAD.

Key Insights and Ideas

  • Pre-trained ViTs are superior for unification: The authors empirically demonstrate that self-supervised ViTs, when used as off-the-shelf encoders, provide a more robust and generalizable foundation for unified anomaly detection compared to supervised ConvNets. This is attributed to their ability to capture richer, more diverse representations without being overly specialized by task-specific labels.
  • Task-agnostic components are crucial: BPM and top k-ratio feature matching are designed to be independent of specific tasks or datasets. This allows UniFormaly to seamlessly adapt to different anomaly detection scenarios without requiring task-specific fine-tuning.
  • Unified anomaly scoring is achievable: The top k-ratio feature matching approach effectively bridges the gap between different anomaly levels (local defects vs. semantic anomalies) by casting anomaly scoring into a multiple instance learning framework. This leads to more consistent performance across diverse tasks.
  • BPM enhances efficiency and robustness: By masking background regions, BPM significantly reduces memory requirements and computational load, making the framework scalable. It also helps in handling both false positives and false negatives.

Key Results

UniFormaly achieves state-of-the-art or highly competitive results across various anomaly detection tasks:

  • Defect Detection: Achieves AUROC of 99.32 on MVTecAD, outperforming existing methods. It also shows significantly less standard deviation across classes, indicating better generalization.
  • Semantic Anomaly Detection: Demonstrates strong performance, particularly on high-resolution datasets like ImageNet-30 and Species-60, with significant performance margins over baselines.
  • Multi-class Anomaly Detection: Shows minimal performance drop in AUROC when extended to multi-class scenarios, outperforming other methods, including UniAD [39].
  • Anomaly Clustering: Outperforms existing methods, including the specialized anomaly clustering method [28], achieving improved F1 scores.
  • Low-shot Anomaly Detection: Achieves competitive AUROC even with limited normal data, improving the state-of-the-art by approximately 8% in one-shot scenarios.

Limitations and Future Work

  • Limitations: The paper notes that while UniFormaly is effective, the performance on lower-resolution datasets like CIFAR-10/100 might be slightly impacted due to ViTs being pre-trained on higher resolutions. The authors also acknowledge that their current work focuses on 2D images.
  • Future Work: The authors suggest that the inherent scalability of the Vision Transformer architecture opens avenues for expanding UniFormaly's task coverage to other modalities, such as sensory data, video data, and natural language. They intend to explore this direction in future research.

Summary updated

Abstract

The core idea of visual anomaly detection is to learn the normality from normal images, but previous works have been developed specifically for certain tasks, leading to fragmentation among various tasks: defect detection, semantic anomaly detection, multi-class anomaly detection, and anomaly clustering. This one-task-one-model approach is resource-intensive and incurs high maintenance costs as the number of tasks increases. This paper presents SelFormaly, a universal and powerful anomaly detection framework. We emphasize the necessity of our off-the-shelf approach by pointing out a suboptimal issue with fluctuating performance in previous online encoder-based methods. In addition, we question the effectiveness of using ConvNets as previously employed in the literature and confirm that self-supervised ViTs are suitable for unified anomaly detection. We introduce back-patch masking and discover the new role of top k-ratio feature matching to achieve unified and powerful anomaly detection. Back-patch masking eliminates irrelevant regions that possibly hinder target-centric detection with representations of the scene layout. The top k-ratio feature matching unifies various anomaly levels and tasks. Finally, SelFormaly achieves state-of-the-art results across various datasets for all the aforementioned tasks.
BibTeX
@article{lee2026uniformaly,
  author = {Yujin Lee and Harin Lim and Seoyoon Jang and Hyunsoo Yoon},
  title = {{UniFormaly: Towards task-agnostic unified framework for visual anomaly detection}},
  journal = {Pattern Recognition},
  year = {2026},
  doi = {10.1016/j.patcog.2025.111820},
}
Figure