Reducing Anomaly Detection in Images to Detection in Noise

Axel Davy, Thibaud Ehret, Jean-Michel Morel, Mauricio Delbracio

Abstract

Anomaly detectors address the difficult problem of detecting automatically exceptions in an arbitrary background image. Detection methods have been proposed by the thousands because each problem requires a different background model. By analyzing the existing approaches, we show that the problem can be reduced to detecting anomalies in residual images (extracted from the target image) in which noise and anomalies prevail. Hence, the general and impossible background modeling problem is replaced by simpler noise modeling, and allows the calculation of rigorous thresholds based on the a contrario detection theory. Our approach is therefore unsupervised and works on arbitrary images.

Introduction

Anomaly detection—identifying the "needle in the haystack" within an image—is a fundamental challenge in computer vision. The primary difficulty lies in the "haystack" itself: the background varies significantly across different scenes, making a universal background model seemingly impossible. Traditional approaches often rely on large training sets of "normal" data or focus on "saliency," which can introduce human-centric semantic biases that are irrelevant for industrial or medical contexts. This paper proposes a paradigm shift: instead of modeling the complex background, we remove it to reveal a residual image where anomalies can be detected as statistical outliers in noise.

Methodology

The proposed method operates on the principle that "normal" content is inherently repetitive within an image. The process involves two main stages:

1. Self-Similarity Removal

For every patch in the image, the algorithm searches for the $n$ most similar patches elsewhere in the same image (excluding the immediate neighborhood). A "normal" version of the patch is then reconstructed by averaging these neighbors. Subtracting this reconstruction from the original image yields a residual image. In this residual, the complex background is effectively neutralized, leaving behind what can be modeled as "colored Gaussian noise" plus any non-repetitive anomalies.

The anomaly detection process. The top row shows input and detections. The bottom rows (a) and (b) illustrate the multi-scale residual images where background patterns are suppressed, leaving anomalies visible against a noise-like background.

2. Statistical Detection via A Contrario Theory

The residual image is analyzed using the a contrario detection framework. This mathematical approach calculates the Number of False Alarms (NFA) for any given structure. If a structure is statistically extremely unlikely to occur by chance in a noise field, it is flagged as an anomaly. This provides a rigorous, mathematically guaranteed threshold for detection, which is crucial for high-stakes applications.

The method is flexible and can be applied to raw pixels or to feature maps from deep networks like VGG-19 to capture more abstract, mid-level structural anomalies.

Experimental Results

The authors validated the "universal" nature of their approach across diverse datasets, including synthetic toy examples, the Toronto saliency dataset, medical mammography, and specialized industrial imagery such as radar and textile defects.

Comparative results across various image types. The proposed method (columns 2-5) consistently identifies anomalies across synthetic, real-world, and specialized domains, often outperforming established saliency and anomaly detection benchmarks (columns 6-9).

Key findings include:

  • High Confidence: In medical imaging, the method detected tumors with an NFA of $10^{-12}$, indicating extremely high statistical confidence compared to previous methods.
  • Robustness: The system successfully identified anomalies—ranging from a red sign on a brick wall to a man in a forest—without any domain-specific re-training.
  • Feature Depth: Utilizing VGG features (layers conv1_1 to conv3_1) enabled the detection of structural anomalies that were not apparent in the raw pixel space.

Discussion and Conclusion

The core insight of this work is that "normal is repetitive." By reducing the impossible task of background modeling to the well-understood problem of noise modeling, the authors provide a truly unsupervised and versatile framework. Unlike deep learning models that produce ambiguous heatmaps, the a contrario approach offers a rigorous statistical basis for decision-making.

While the current method excels at low-to-mid-level patterns, future work may address high-level semantic anomalies that might be locally self-similar. Additionally, extending this framework to the temporal domain by analyzing motion fields in video presents a promising avenue for further research.

Summary updated

Abstract

Anomaly detectors address the difficult problem of detecting automatically exceptions in an arbitrary background image. Detection methods have been proposed by the thousands because each problem requires a different background model. By analyzing the existing approaches, we show that the problem can be reduced to detecting anomalies in residual images (extracted from the target image) in which noise and anomalies prevail. Hence, the general and impossible background modeling problem is replaced by simpler noise modeling, and allows the calculation of rigorous thresholds based on the a contrario detection theory. Our approach is therefore unsupervised and works on arbitrary images.
BibTeX
@inproceedings{davy2018reducing,
  author = {Axel Davy and Thibaud Ehret and Jean-Michel Morel and Mauricio Delbracio},
  title = {{Reducing Anomaly Detection in Images to Detection in Noise}},
  booktitle = {ICIP},
  year = {2018},
  url = {https://openreview.net/forum?id=Irkwwv8121},
}
Figure