TL;DR
This paper introduces two new challenging datasets, CARS-AD and ROADS-AD, for unsupervised anomaly detection, featuring significant intra-class variability. They demonstrate that current state-of-the-art methods struggle on these heterogeneous datasets, highlighting the need for more robust anomaly detection techniques.
What Problem Does This Paper Address?
Unsupervised anomaly detection (AD) is crucial for identifying unusual patterns in data, with applications ranging from manufacturing quality control to infrastructure monitoring. A key challenge in AD is accurately modeling "normality" when only normal data is available for training. While significant progress has been made, most existing AD benchmarks, like MVTec-AD, consist of highly homogeneous data where objects within a class are very similar. This homogeneity makes it easier for models to learn normality. However, real-world scenarios often involve much more diverse data. For instance, cars can have various colors, models, and lighting conditions, and roads can differ greatly due to climate, terrain, and construction practices across different countries. This paper addresses the gap in evaluating AD methods on such heterogeneous datasets, which are more representative of real-world complexity and pose a greater challenge for anomaly detection.
Methods
The authors evaluate several state-of-the-art unsupervised AD methods, including:
- EfficientAD: A student-teacher architecture that predicts features on healthy samples.
- Reverse Distillation: A variation of student-teacher learning that reproduces multi-scale teacher features.
- CFA: A feature adaptation strategy to prevent overestimation of normality from abnormal features.
- DRÆM: A cascaded reconstructive and discriminative network.
- GANomaly: A GAN-based approach using an encoder-decoder-encoder structure.
- Csflow: Processes multi-scale feature maps jointly using cross-scale normalizing flows for likelihood estimation and localization.
- U-Flow: A model with a multi-scale feature extractor and a U-shaped normalizing flow.
- Fastflow: Maps input images to a normal distribution using normalizing flows.
- DSR: Synthesizes anomalies in a quantized feature space.
The paper leverages the Anomalib library for conducting experiments and uses the AUROC metric for evaluating both image-level anomaly detection and pixel-level anomaly localization.
Datasets
The paper introduces two novel benchmark datasets:
- CARS-AD: Contains real images of cars with various defects, including "broken_part", "dent", "scratch", and "wind_screen". It features significant intra-class variability in car types, colors, and defect appearances.
- ROADS-AD: Consists of real road images from multiple countries (China, Czech Republic, India, Japan, Norway, USA). It highlights challenges arising from differences in climate, terrain, infrastructure, and cultural preferences, leading to diverse road appearances and anomalies like cracks and surface discontinuities.
Both datasets provide pixel-wise ground truth masks for anomaly localization and image-level labels for anomaly detection. They are made publicly available.
Key Insights and Ideas
- Heterogeneity is the new frontier: The paper strongly suggests that current AD benchmarks are too homogeneous, and real-world challenges lie in dealing with significant variations within normal classes.
- Current methods struggle: State-of-the-art AD methods, which perform exceptionally well on datasets like MVTec-AD, show considerably lower performance on CARS-AD and ROADS-AD, indicating a significant gap.
- Multi-scale processing is key for CARS-AD: Methods like Csflow and U-Flow, which effectively process multi-scale features, excel on the CARS-AD dataset. This is attributed to the varying scales of defects and complex image structures.
- Reverse Distillation shines in localization on ROADS-AD: This method demonstrates superior performance in anomaly localization on the ROADS-AD dataset, suggesting its architecture is adept at handling diverse, geographically influenced anomalies.
- No single method dominates all categories: The results show that different methods have strengths and weaknesses across various defect types and countries, emphasizing the need for specialized or more generalized approaches.
Key Results
- Performance drop: All evaluated methods showed significantly lower AUROC scores on CARS-AD and ROADS-AD compared to their performance on MVTec-AD (where AUROC often exceeds 99%).
- CARS-AD: Csflow and U-Flow achieved the best overall performance, with AUROC scores around 0.91/0.69 and 0.81/0.74 (image/pixel level) respectively.
- ROADS-AD: No single method consistently dominated. Reverse Distillation showed superior performance in anomaly localization (pixel level AUROC), with scores up to 0.90 in some categories. Csflow and U-Flow also performed well.
Limitations and Future Work
- Limitations: The paper acknowledges that the performance of all methods is considerably lower on their proposed datasets, highlighting the limitations of current AD techniques when faced with significant intra-class variability and complex real-world scenarios. The heterogeneity of the datasets themselves is the primary challenge.
- Future Work: The authors hope that CARS-AD and ROADS-AD will inspire further research and the development of innovative AD methods capable of handling diverse anomaly types and complex image structures with greater accuracy and reliability. They aim to motivate the community to develop new techniques to tackle these challenges.
Summary updated