Abstract
This paper presents a fast, principled approach for detecting anomalous and out-of-distribution (OOD) samples in deep neural networks (DNN). We propose the application of linear statistical dimensionality reduction techniques on the semantic features produced by a DNN, in order to capture the low-dimensional subspace truly spanned by said features. We show that the "feature reconstruction error" (FRE), which is the $\ell_2$-norm of the difference between the original feature in the high-dimensional space and the pre-image of its low-dimensional reduced embedding, is highly effective for OOD and anomaly detection. To generalize to intermediate features produced at any given layer, we extend the methodology by applying nonlinear kernel-based methods. Experiments using standard image datasets and DNN architectures demonstrate that our method meets or exceeds best-in-class quality performance, but at a fraction of the computational and memory cost required by the state of the art. It can be trained and run very efficiently, even on a traditional CPU.
Introduction
Deep neural networks (DNNs) are increasingly deployed in safety-critical environments, from autonomous vehicles to industrial quality control. However, these models are notoriously overconfident; when presented with out-of-distribution (OOD) data—inputs they were never trained to handle—they often produce high-confidence, yet entirely incorrect, predictions.
Existing solutions for OOD and anomaly detection typically fall into two categories: simple but weak methods (such as basic softmax scores) or powerful but computationally heavy methods (such as ensembles, Bayesian networks, or generative models). The latter often require massive memory and specialized GPU hardware to run in real-time. This paper introduces a third path: a fast, principled method that leverages the low-dimensional subspaces of DNN features to identify anomalies with state-of-the-art accuracy and minimal computational overhead.
Methodology: Subspace Modeling and FRE
The core of the proposed approach is the "manifold hypothesis," which suggests that while DNN features reside in high-dimensional spaces, they actually occupy a much smaller, lower-dimensional manifold. By capturing this subspace, we can determine if a new input aligns with the patterns learned during training.
Feature Reconstruction Error (FRE)
The detection process follows three primary steps:
- Feature Extraction: Features are extracted from a pre-trained DNN at a specific intermediate or semantic layer.
- Dimensionality Reduction: We apply Principal Component Analysis (PCA) for linear subspaces or Kernel PCA (kPCA) for non-linear manifolds. This maps the high-dimensional features into a lower-dimensional embedding.
- Metric Calculation: The Feature Reconstruction Error (FRE) is defined as the $\ell_2$-norm of the difference between the original high-dimensional feature and the pre-image of its low-dimensional embedding.
A low FRE indicates that the sample is well-represented by the training distribution's subspace. Conversely, a high FRE suggests the sample is an anomaly or OOD.
Linear vs. Non-linear Modeling
The research indicates that the choice of dimensionality reduction technique should depend on the layer depth. Standard PCA is highly effective for "semantic" layers near the output, where features tend to be more Gaussian and linear. However, deeper intermediate layers often exhibit complex non-linearity, making kPCA (specifically using RBF kernels) a more suitable choice for capturing the feature manifold.
Experimental Setup and Results
The methodology was evaluated across two distinct tasks: OOD detection and industrial anomaly detection.
Datasets
- OOD Detection: Evaluated using CIFAR-10, CIFAR-100, and SVHN as in-distribution data, with LSUN and resized variants as OOD sets.
- Anomaly Detection: Tested on the MVTec industrial inspection dataset (covering defects in materials like carpets and leather) and the Magnetic Tile (MTD) dataset.
Performance and Efficiency
The proposed method achieved an average AUROC of 98.4% on the MVTec dataset, nearly matching the performance of the state-of-the-art PatchCore (99.1%) while requiring significantly lower computational complexity. On the Magnetic Tile dataset, the method achieved a 99.2% AUROC, outperforming specialized existing methods.
Beyond accuracy, the method's efficiency is its primary advantage. Because it relies on fundamental linear algebra (SVD and dot products), it can be trained and executed on traditional CPUs. Furthermore, the system is remarkably data-frugal; performance remains robust even when the training data used to build the subspaces is reduced to only 20% of the original set.
Discussion and Insights
Several key insights emerged from the study:
- Redundancy in DNNs: The paper confirms that DNN features are highly redundant. For example, in a ResNet-18 layer with 512 dimensions, approximately 99.5% of the information can be captured in just 29 dimensions.
- Per-Class Modeling: Building separate subspaces for each category (per-class modeling) significantly outperforms global modeling for OOD tasks.
- Hardware Accessibility: By avoiding complex probabilistic modeling and generative training, this approach makes high-performance anomaly detection accessible for "edge" devices in industrial settings where GPUs may not be available.
Limitations and Future Work
While the current method is highly effective using single layers, future research could explore automated layer selection to optimize performance for specific tasks. Additionally, while the RBF kernel was the primary focus for kPCA, exploring other kernels or automated tuning could further enhance results for deeper network layers. Finally, the fusion of reconstruction errors from multiple layers simultaneously may provide a more comprehensive "uncertainty profile" for complex inputs.
Summary updated