Probabilistic Modeling of Deep Features for Out-of-Distribution and Adversarial Detection

Nilesh A. Ahuja, Ibrahima J. Ndiour, Trushant Kalyanpur, Omesh TickooarXiv 2019 · OpenReview

Abstract

We present a principled approach for detecting out-of-distribution (OOD) and adversarial samples in deep neural networks (DNNs). By modeling the internal feature distributions of a trained network using parametric probability distributions—specifically Gaussian and Gaussian Mixture Models (GMMs)—we derive uncertainty estimates that reliably discriminate in-distribution samples from anomalies. Our methodology demonstrates significant improvements in detection metrics (up to 12 percentage points in AUPR/AUROC) and provides the first reported reliable detection of white-box adversarial framing in video classifiers.

The Challenge of Neural Network Overconfidence

Deep neural networks are notoriously overconfident. Standard architectures often produce high-confidence predictions even when presented with inputs that look nothing like their training data (Out-of-Distribution) or inputs specifically engineered to fool them (Adversarial). The standard Softmax output provides only a relative ranking of classes, lacking an absolute measure of certainty. While Bayesian neural networks offer a solution, their high computational cost makes them impractical for many real-world applications. This research seeks a more efficient, "post-hoc" method to extract reliable uncertainty scores from standard, pre-trained networks.

A Generative Approach to Deep Features

Instead of relying solely on the final output layer, our approach analyzes "deep features"—the internal activations at various layers of the network. Once a network is trained, we pass the training data through it to fit parametric probability distributions to the features of each class.

Visualization of class-specific feature distributions. The scatter plot (left) shows distinct clustering of two labels, while the density map (right) illustrates the learned probabilistic representation.

To manage the high dimensionality of these features, we employ average pooling and Principal Component Analysis (PCA) to compress the data before modeling. During inference, we calculate the log-likelihood of a new sample's features against these learned distributions. A low likelihood serves as a "tell-tale" sign that the sample is OOD or adversarial.

Beyond Simple Gaussians: The GMM Advantage

Previous research often assumed that deep features follow a multivariate Gaussian distribution with a shared (tied) covariance matrix across all classes. We argue that this assumption is overly restrictive. Our work explores more flexible models:

  • Separate Covariance: Each class is assigned its own unique Gaussian distribution.
  • Gaussian Mixture Models (GMMs): Each class is modeled as a combination of multiple Gaussians, allowing for the capture of complex, non-convex shapes in the feature space.
Comparison between a single 2D Gaussian (right) and a 10-component GMM (left) modeling a ring-like feature distribution. The GMM captures the underlying data manifold far more accurately.

As shown in Figure 6, a single Gaussian fails to capture the structure of complex feature manifolds, whereas a GMM provides a much more faithful representation. This precision is critical for internal layers where the data has not yet been linearized by the network.

Experimental Validation and Results

The methodology was validated across image (MNIST, CIFAR10) and video (UCF101) domains.

  • Detection Performance: Switching from tied-covariance Gaussians to GMMs or separate-covariance models resulted in improvements of up to 12 percentage points in AUPR and AUROC for OOD detection.
  • Classification Parity: Probabilistic models used for classification yielded accuracy comparable to standard Softmax, confirming they are faithful representations of the network's logic.
  • Video Adversarial Detection: Our approach is the first to reliably detect white-box adversarial framing in video classifiers. While these attacks cause accuracy to plummet, the internal feature likelihoods remain a robust indicator of the attack.

Conclusion

Modeling deep features with flexible parametric distributions provides a powerful and efficient mechanism for anomaly detection in DNNs. While GMMs increase computational complexity, the significant gains in detection reliability justify their use, especially in safety-critical applications. Future work will explore how these distributions evolve during training and the use of large-scale datasets to inform prior distributions for specialized tasks.

Summary updated

Abstract

We present a principled approach for detecting out-of-distribution (OOD) and adversarial samples in deep neural networks. Our approach consists in modeling the outputs of the various layers (deep features) with parametric probability distributions once training is completed. At inference, the likelihoods of the deep features w.r.t the previously learnt distributions are calculated and used to derive uncertainty estimates that can discriminate in-distribution samples from OOD samples. We explore the use of two classes of multivariate distributions for modeling the deep features - Gaussian and Gaussian mixture - and study the trade-off between accuracy and computational complexity. We demonstrate benefits of our approach on image features by detecting OOD images and adversarially-generated images, using popular DNN architectures on MNIST and CIFAR10 datasets. We show that more precise modeling of the feature distributions result in significantly improved detection of OOD and adversarial samples; up to 12 percentage points in AUPR and AUROC metrics. We further show that our approach remains extremely effective when applied to video data and associated spatio-temporal features by detecting adversarial samples on activity classification tasks using UCF101 dataset, and the C3D network. To our knowledge, our methodology is the first one reported for reliably detecting white-box adversarial framing, a state-of-the-art adversarial attack for video classifiers.
BibTeX
@misc{ahuja2019probabilistic,
  author = {Nilesh A. Ahuja and Ibrahima J. Ndiour and Trushant Kalyanpur and Omesh Tickoo},
  title = {{Probabilistic Modeling of Deep Features for Out-of-Distribution and Adversarial Detection}},
  year = {2019},
  eprint = {1909.11786},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.11786},
}
Figure