A Comparison of Supervised and Unsupervised Deep Learning Methods for Anomaly Detection in Images

Vincent Wilmet, Sauraj Verma, Tabea Redl, Håkon Sandaker, Zhenning LiarXiv 2021 · arXiv

Introduction

In industrial manufacturing, identifying defects—anomalies—is a critical component of quality assurance. While human inspectors are proficient at identifying irregularities such as scratches, missing components, or surface discolorations, automating this process using artificial intelligence presents significant challenges. The primary obstacle is extreme class imbalance: while "normal" production samples are abundant, anomalous samples are rare and highly variable. This study evaluates whether supervised learning, which requires labeled examples of defects, or unsupervised learning, which models the distribution of normal data, is more effective for industrial anomaly detection.

Methodology

The researchers evaluated three primary architectural strategies to address the anomaly detection task. The first is a Supervised CNN, a standard classification network trained to distinguish between "normal" and "anomalous" classes. The second is the Kernel Density Convolutional Autoencoder (KD-CAE), an unsupervised model that learns to compress and reconstruct images. By training exclusively on normal data, the CAE is expected to fail when reconstructing anomalous features, creating a measurable "reconstruction error." This error is combined with Kernel Density Estimation (KDE) to identify outliers.

High-level overview of the experimental pipeline, comparing supervised (CNN) and unsupervised (AE, GAN) approaches.

Additionally, the study explored a Noise Injected CAE (NI-CAE) to test if Gaussian noise during training improves robustness, and a Deep Convolutional Generative Adversarial Network (DCGAN) to generate synthetic training data for sparse datasets, such as the Toothbrush category.

Experimental Setup

The evaluation utilized the MVTec Anomaly Detection (AD) dataset, a comprehensive benchmark containing 15 categories of industrial objects and textures. To manage computational requirements, high-resolution images were downsampled to 256x256 or 128x128 pixels. Performance was measured using F1 scores to account for the inherent class imbalance in the test sets.

Results and Discussion

The experimental results revealed a stark performance gap between supervised and unsupervised approaches. The supervised CNN performed poorly across all categories, with F1 scores often falling below 0.25. This failure underscores the "data hunger" of supervised models; without a vast and representative library of possible defects, the model cannot generalize effectively to identify anomalies it has never seen.

In contrast, the unsupervised KD-CAE achieved significantly higher performance, reaching F1 scores of 0.85 for the Toothbrush category and 0.87 for Bottles. This success is attributed to the Overfitting Paradox: by intentionally overfitting the autoencoder to the "normal" distribution, the model becomes incapable of reconstructing anomalous features. When an anomaly is processed, the resulting high reconstruction error serves as a reliable signal for detection.

Original vs. reconstructed images for various MVTec AD categories using the Convolutional Autoencoder.

However, the unsupervised models faced difficulties with specific datasets, notably "Screws." Because the screws appeared in varying orientations and positions, the autoencoder struggled to define a consistent "normal" baseline, leading to high reconstruction errors even for non-defective samples. Furthermore, while noise injection (NI-CAE) was hypothesized to improve robustness, it generally degraded performance compared to the standard KD-CAE, except in the case of the Transistor dataset.

Conclusion

This research demonstrates that unsupervised methods, specifically Convolutional Autoencoders, are superior to traditional supervised CNNs for industrial anomaly detection due to their ability to learn from unlabeled, "normal-only" data. While challenges remain regarding spatial invariance and the detection of extremely subtle defects, the unsupervised approach bypasses the prohibitive cost and difficulty of collecting large-scale anomalous datasets. Future research should investigate architectures like AnoGAN to further refine the use of generative models in identifying outliers within complex latent spaces.

Summary updated

Abstract

Anomaly detection in images plays a significant role for many applications across all industries, such as disease diagnosis in healthcare or quality assurance in manufacturing. Manual inspection of images, when extended over a monotonously repetitive period of time is very time consuming and can lead to anomalies being overlooked.Artificial neural networks have proven themselves very successful on simple, repetitive tasks, in some cases even outperforming humans. Therefore, in this paper we investigate different methods of deep learning, including supervised and unsupervised learning, for anomaly detection applied to a quality assurance use case. We utilize the MVTec anomaly dataset and develop three different models, a CNN for supervised anomaly detection, KD-CAE for autoencoder anomaly detection, NI-CAE for noise induced anomaly detection and a DCGAN for generating reconstructed images. By experiments, we found that KD-CAE performs better on the anomaly datasets compared to CNN and NI-CAE, with NI-CAE performing the best on the Transistor dataset. We also implemented a DCGAN for the creation of new training data but due to computational limitation and lack of extrapolating the mechanics of AnoGAN, we restricted ourselves just to the generation of GAN based images. We conclude that unsupervised methods are more powerful for anomaly detection in images, especially in a setting where only a small amount of anomalous data is available, or the data is unlabeled.
BibTeX
@misc{wilmet2021comparison,
  author = {Vincent Wilmet and Sauraj Verma and Tabea Redl and Håkon Sandaker and Zhenning Li},
  title = {{A Comparison of Supervised and Unsupervised Deep Learning Methods for Anomaly Detection in Images}},
  year = {2021},
  eprint = {2107.09204},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.09204},
}
Figure