Energy-Based Anomaly Detection and Localization

Ergin Utku Genc, Nilesh A. Ahuja, Ibrahima J. Ndiour, Omesh TickooarXiv 2021 · OpenReview

Abstract

This paper presents initial progress towards a unified energy-based solution for semi-supervised visual anomaly detection and localization. Using only anomaly-free training data, the proposed method employs density estimates from an Energy-Based Model (EBM) as normalcy scores to discriminate between normal and anomalous images. Furthermore, by back-propagating the gradients of the energy score with respect to the input image, the model generates gradient maps for pixel-level spatial localization. Experiments on the MVTec industrial dataset demonstrate that simple processing of these gradient maps can provide normalcy scores that match or exceed detection performance obtained from raw energy values, highlighting the potential of EBMs for unforeseen anomaly detection.

Introduction

In industrial manufacturing, identifying defects is a critical task. However, while "normal" data is abundant, "anomalous" data—such as specific cracks, structural deformations, or foreign contaminants—is rare and highly unpredictable. This scarcity makes traditional supervised learning approaches impractical. Consequently, semi-supervised methods that learn the distribution of normal data are preferred.

While many existing methods focus solely on detection (binary classification of an image as good or bad), localization (identifying the specific region of the defect) is equally vital for diagnostic purposes. Current localization techniques often rely on reconstruction-based models like Variational Autoencoders (VAEs) or Generative Adversarial Networks (GANs). These models can be difficult to train and may fail to capture subtle textures. This research explores whether Energy-Based Models (EBMs) can provide a more robust, unified solution for both detection and localization.

Methodology

The core of the proposed approach is treating an EBM as a "normalcy" estimator. The model learns an energy function $E(x)$, where low energy corresponds to high-probability (normal) data.

Detection and Localization via Gradients

During inference, a high energy score flags an image as a potential anomaly. To localize the anomaly, the authors back-propagate the gradient of the energy score with respect to the input image pixels. This process creates a "gradient map" where high-magnitude regions indicate areas contributing most to the "abnormality" of the image.

Gradient Standardization

A significant challenge is that raw gradient maps are often noisy; edges or complex textures in normal images naturally produce higher gradients. To address this, the authors implement a standardization step. They calculate the mean and standard deviation for every pixel across the entire training set. A standardized score (z-score) is then computed for test pixels, highlighting only those deviations that are statistically significant compared to the training baseline.

Standardization of gradient maps and score distributions. The standardized scores (bottom right) show a much clearer separation between 'Good' and 'Defective' samples compared to raw scores (top right).

The model is trained using a Maximum Likelihood approach with Contrastive Divergence (CD) and a Stochastic Gradient Langevin Dynamics (SGLD) sampler to generate negative samples.

Experimental Results

The method was evaluated using the MVTec AD (Anomaly Detection) dataset, which consists of 15 categories of industrial objects and textures. The model was trained exclusively on defect-free images and tested on images containing unseen defects like scratches or holes.

Performance Metrics

The EBM achieved an average AUROC (Area Under the Receiver Operating Characteristic) of 0.72 for detection across all categories. While this trailed specialized benchmarks, it represents a strong preliminary result for an EBM-based approach. Localization performance was particularly strong in texture categories; for instance, the model achieved localization AUROCs of 0.86 on "Grid" and 0.87 on "Leather."

Visual localization of anomalies across different MVTec categories. The heatmaps generated from energy gradients align closely with the ground truth defect masks.

Standardization proved essential, improving the average detection AUROC from 0.69 to 0.72 and providing much cleaner spatial heatmaps.

Conclusion and Future Work

This research demonstrates that the gradient of an EBM's energy score carries interpretable spatial information regarding anomalies. By modeling the distribution of gradients at each pixel, the system can effectively filter out expected noise in complex images.

However, several challenges remain. EBMs are computationally expensive and can be unstable to train, often requiring downsampled images (e.g., 128x128) which may obscure fine-scale defects. Future work will focus on improving training stability and increasing model capacity to close the performance gap with current state-of-the-art industrial inspection models.

Summary updated

Abstract

This brief sketches initial progress towards a unified energy-based solution for the semi-supervised visual anomaly detection and localization problem. In this setup, we have access to only anomaly-free training data and want to detect and identify anomalies of an arbitrary nature on test data. We employ the density estimates from the energy-based model (EBM) as normalcy scores that can be used to discriminate normal images from anomalous ones. Further, we back-propagate the gradients of the energy score with respect to the image in order to generate a gradient map that provides pixel-level spatial localization of the anomalies in the image. In addition to the spatial localization, we show that simple processing of the gradient map can also provide alternative normalcy scores that either match or surpass the detection performance obtained with the energy value. To quantitatively validate the performance of the proposed method, we conduct experiments on the MVTec industrial dataset. Though still preliminary, our results are very promising and reveal the potential of EBMs for simultaneously detecting and localizing unforeseen anomalies in images.
BibTeX
@misc{genc2021energy,
  author = {Ergin Utku Genc and Nilesh A. Ahuja and Ibrahima J. Ndiour and Omesh Tickoo},
  title = {{Energy-Based Anomaly Detection and Localization}},
  year = {2021},
  eprint = {2105.03270},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.03270},
}
Figure