Generative probabilistic novelty detection with adversarial autoencoders

Donald AdjerohOpenReview 2022 · OpenReview

TL;DR

This paper introduces Generative Probabilistic Novelty Detection (GPND), a novel approach for identifying outliers by directly estimating the probability of a data point belonging to the inlier distribution. By combining adversarial autoencoders with a probabilistic framework that linearizes the data manifold, GPND achieves state-of-the-art performance on several benchmark datasets.

What Problem Does This Paper Address?

Novelty detection, the task of distinguishing between "normal" (inlier) and "abnormal" (outlier) data points, is a critical problem in many domains, from medical diagnostics to industrial fault detection. A particularly challenging scenario is when only inlier data is available for training, and outliers are rare and diverse. Existing deep learning methods often rely on reconstruction errors from autoencoders or train one-class classifiers. However, these methods can be indirect or struggle with the inherent rarity and variability of outliers. This paper aims to address this by directly modeling the inlier distribution and calculating the probability of a new data point belonging to it.

Methods

The core of GPND lies in its probabilistic approach to novelty detection, leveraging adversarial autoencoders (AAEs). Here's a breakdown of the key techniques:

  • Generative Probabilistic Approach: Instead of relying solely on reconstruction error, GPND directly computes the probability that a test sample was generated by the inlier distribution. This is achieved by modeling the underlying data manifold.
  • Adversarial Autoencoders (AAEs): The authors use AAEs to learn a mapping (encoder $g$) from the data space to a latent space, and a reverse mapping (decoder $f$) from the latent space back to the data space. The "adversarial" aspect comes from using discriminators to ensure the latent space distribution matches a prior (e.g., a standard normal distribution) and that the decoder can generate realistic inlier-like data.
  • Manifold Linearization: To make the probability computation tractable, the paper linearizes the learned data manifold around a test point. This allows the probability density to be factorized with respect to local coordinates in the manifold's tangent space and its orthogonal complement.
  • Factorized Probability: The novelty score is computed by combining two components: one related to the inlier distribution on the manifold (learned by the decoder and latent distribution) and another related to the noise (orthogonal to the manifold). This factorization simplifies computation.
  • Improved Training: The AAE training is enhanced with an additional adversarial component that matches the decoder's output distribution to the inlier data distribution, reducing blurriness and improving local details in generated images.

Datasets

The authors evaluate GPND on several standard datasets:

  • MNIST: A dataset of handwritten digits, used for evaluating novelty detection by treating one digit as the inlier class and others as outliers.
  • Coil-100: A dataset of object images, used with varying numbers of inlier categories and randomly chosen outliers.
  • Fashion-MNIST: Similar to MNIST but with fashion product images, used to test the approach on a different domain.
  • CIFAR-10 / CIFAR-100: Datasets of small natural images, used for comparison with other methods, particularly ODIN.

Key Insights and Ideas

  • Direct Probability is Key: The paper argues that directly modeling the inlier probability distribution is more effective for novelty detection than relying solely on reconstruction error, which can be an indirect measure.
  • Manifold Structure Matters: Learning the underlying manifold of the inlier data is crucial for accurately estimating its probability distribution.
  • Linearization for Tractability: The clever use of manifold linearization makes the complex probabilistic calculations computationally feasible for practical novelty detection.
  • Adversarial Training for Quality: Incorporating adversarial losses not only for the latent space but also for the generated data significantly improves the quality of reconstructions and the overall model.

Key Results

GPND demonstrates strong performance across various datasets and metrics:

  • State-of-the-Art Performance: The paper reports state-of-the-art results on several benchmarks, including MNIST, Coil-100, and Fashion-MNIST, often outperforming existing methods in terms of F1-score and AUROC.
  • Competitive with ODIN: On CIFAR-10/100, GPND achieves competitive results with ODIN, a method that relies on pre-trained classifiers, and even outperforms ODIN on CIFAR-100, suggesting robustness to increased class complexity.
  • Ablation Studies: Experiments show that each component of the GPND model (latent distribution matching, data distribution matching, and reconstruction loss) plays a significant role in achieving the best performance.

Limitations and Future Work

  • Computational Cost: While the approach is made computationally feasible, computing derivatives and SVDs for each test sample can still be time-consuming, though batch processing helps.
  • Hyperparameter Tuning: The paper mentions that latent space size was chosen to maximize F1 on validation sets, indicating that hyperparameter tuning is important for optimal performance.
  • Future Work: The paper doesn't explicitly detail future work directions, but the success of GPND suggests potential applications in other domains requiring robust anomaly detection and further exploration of its generative capabilities.

Summary updated

Abstract

Novelty detection is the problem of identifying whether a new data point is considered to be an inlier or an outlier. We assume that training data is available to describe only the inlier distribution. Recent approaches primarily leverage deep encoder-decoder network architectures to compute a reconstruction error that is used to either compute a novelty score or to train a one-class classifier. While we too leverage a novel network of that kind, we take a probabilistic approach and effectively compute how likely it is that a sample was generated by the inlier distribution. We achieve this with two main contributions. First, we make the computation of the novelty probability feasible because we linearize the parameterized manifold capturing the underlying structure of the inlier distribution, and show how the probability factorizes and can be computed with respect to local coordinates of the manifold tangent space. Second, we improve the training of the autoencoder network. An extensive set of results show that the approach achieves state-of-the-art performance on several benchmark datasets.
BibTeX
@misc{adjeroh2022generative,
  author = {Donald Adjeroh},
  title = {{Generative probabilistic novelty detection with adversarial autoencoders}},
  year = {2022},
  url = {https://openreview.net/forum?id=KvP-CXgStH},
  howpublished = {OpenReview},
}

Loading related content...

Figure