Abstract
This paper presents initial progress towards a unified energy-based solution for semi-supervised visual anomaly detection and localization. Using only anomaly-free training data, the proposed method employs density estimates from an Energy-Based Model (EBM) as normalcy scores to discriminate between normal and anomalous images. Furthermore, by back-propagating the gradients of the energy score with respect to the input image, the model generates gradient maps for pixel-level spatial localization. Experiments on the MVTec industrial dataset demonstrate that simple processing of these gradient maps can provide normalcy scores that match or exceed detection performance obtained from raw energy values, highlighting the potential of EBMs for unforeseen anomaly detection.
Introduction
In industrial manufacturing, identifying defects is a critical task. However, while "normal" data is abundant, "anomalous" data—such as specific cracks, structural deformations, or foreign contaminants—is rare and highly unpredictable. This scarcity makes traditional supervised learning approaches impractical. Consequently, semi-supervised methods that learn the distribution of normal data are preferred.
While many existing methods focus solely on detection (binary classification of an image as good or bad), localization (identifying the specific region of the defect) is equally vital for diagnostic purposes. Current localization techniques often rely on reconstruction-based models like Variational Autoencoders (VAEs) or Generative Adversarial Networks (GANs). These models can be difficult to train and may fail to capture subtle textures. This research explores whether Energy-Based Models (EBMs) can provide a more robust, unified solution for both detection and localization.
Methodology
The core of the proposed approach is treating an EBM as a "normalcy" estimator. The model learns an energy function $E(x)$, where low energy corresponds to high-probability (normal) data.
Detection and Localization via Gradients
During inference, a high energy score flags an image as a potential anomaly. To localize the anomaly, the authors back-propagate the gradient of the energy score with respect to the input image pixels. This process creates a "gradient map" where high-magnitude regions indicate areas contributing most to the "abnormality" of the image.
Gradient Standardization
A significant challenge is that raw gradient maps are often noisy; edges or complex textures in normal images naturally produce higher gradients. To address this, the authors implement a standardization step. They calculate the mean and standard deviation for every pixel across the entire training set. A standardized score (z-score) is then computed for test pixels, highlighting only those deviations that are statistically significant compared to the training baseline.
The model is trained using a Maximum Likelihood approach with Contrastive Divergence (CD) and a Stochastic Gradient Langevin Dynamics (SGLD) sampler to generate negative samples.
Experimental Results
The method was evaluated using the MVTec AD (Anomaly Detection) dataset, which consists of 15 categories of industrial objects and textures. The model was trained exclusively on defect-free images and tested on images containing unseen defects like scratches or holes.
Performance Metrics
The EBM achieved an average AUROC (Area Under the Receiver Operating Characteristic) of 0.72 for detection across all categories. While this trailed specialized benchmarks, it represents a strong preliminary result for an EBM-based approach. Localization performance was particularly strong in texture categories; for instance, the model achieved localization AUROCs of 0.86 on "Grid" and 0.87 on "Leather."
Standardization proved essential, improving the average detection AUROC from 0.69 to 0.72 and providing much cleaner spatial heatmaps.
Conclusion and Future Work
This research demonstrates that the gradient of an EBM's energy score carries interpretable spatial information regarding anomalies. By modeling the distribution of gradients at each pixel, the system can effectively filter out expected noise in complex images.
However, several challenges remain. EBMs are computationally expensive and can be unstable to train, often requiring downsampled images (e.g., 128x128) which may obscure fine-scale defects. Future work will focus on improving training stability and increasing model capacity to close the performance gap with current state-of-the-art industrial inspection models.
Summary updated