TransductGAN: a Transductive Adversarial Model for Novelty Detection

Najiba Toron, Janaina Mourão Miranda, John Shawe-TaylorarXiv 2022 · DOI · arXiv

Abstract

Novelty detection is a critical task in machine learning, typically addressed through inductive methods trained solely on "normal" data. However, these methods often struggle with hyperparameter tuning due to the absence of outlier examples during training. This paper discusses TransductGAN, a transductive approach that utilizes unlabeled test data—which contains a mixture of normal and novel instances—to learn the characteristics of the unknown. By employing an Adversarial Autoencoder (AAE) with a dual-Gaussian latent space and a Generative Adversarial Network (GAN), TransductGAN generates synthetic novel examples. This allows the novelty detection problem to be reframed as a supervised binary classification task. Our evaluation on MNIST and CIFAR-10 benchmarks demonstrates that TransductGAN achieves state-of-the-art performance, particularly as the contamination rate of the test set increases.

1. Introduction

Novelty detection involves identifying observations that deviate from the distribution of data seen during training. Traditionally, this is treated as a "one-class" problem where a model learns the boundary of "normality." A significant limitation of this inductive paradigm is the difficulty of model selection and hyperparameter tuning; without examples of what constitutes a "novelty," it is nearly impossible to optimize the decision threshold effectively.

TransductGAN addresses this by shifting to a transductive framework. In many real-world applications, such as forensic analysis or batch processing of sensor data, an unlabeled set of test data is available at training time. This test set is usually "contaminated" with the very novelties the system aims to detect. TransductGAN leverages this contamination to infer the distribution of the novel class.

2. Methodology

The TransductGAN architecture integrates an Adversarial Autoencoder (AAE) with a Generative Adversarial Network (GAN) to bridge the gap between latent representations and image synthesis.

2.1 Latent Space Modeling

The AAE maps input data into a low-dimensional latent space. Unlike standard AAEs that regularize the latent space toward a single distribution (e.g., a standard Gaussian), TransductGAN utilizes a mixture of two Gaussians. One component represents the "normal" class, while the other represents the "novel" class. By training on both the labeled normal data and the unlabeled (mixed) test data, the encoder learns to project inputs into these distinct regions of the latent space.

2.2 Synthetic Novelty Generation

The GAN component, specifically a Wasserstein GAN with Gradient Penalty (WGAN-GP) for improved stability, uses the latent distributions to generate synthetic images. The generator learns to map the "novel" Gaussian distribution in the latent space to realistic images that resemble the anomalies present in the unlabeled test set. Remarkably, the model can generate recognizable images of classes it has never seen in a labeled format.

2.3 Binary Classification

Once the generative process is complete, the model produces a balanced dataset of 5,000 synthetic "normal" images and 5,000 synthetic "novel" images. A standard binary classifier, such as a Support Vector Machine (SVM), is then trained on the features extracted from these synthetic images. This converts the unsupervised novelty detection problem into a supervised classification task, providing a clear decision boundary.

3. Experimental Results

The researchers evaluated TransductGAN against leading inductive models, including GANomaly and EGBAD, using the MNIST and CIFAR-10 datasets.

3.1 Performance on MNIST

On the MNIST handwritten digit dataset, TransductGAN showed a significant performance advantage. At a 5% contamination rate (where 5% of the test set consists of a "novel" digit), TransductGAN achieved an Area Under the Receiver Operating Characteristic (AUROC) curve of 0.956, far surpassing GANomaly's 0.783. As the contamination rate increased to 30%, the model's AUROC improved to 0.994, indicating that more unlabeled data allows the model to better characterize the novel class.

3.2 Performance on CIFAR-10

CIFAR-10 represents a more complex challenge involving real-world objects. Even in this high-variance environment, TransductGAN outperformed its peers. At a 5% contamination rate, it achieved an AUROC of 0.659, compared to 0.611 for GANomaly.

4. Discussion and Limitations

The primary insight of TransductGAN is that synthetic outliers, even if imperfect, are significantly more useful for training a detector than no outliers at all. By "peeking" at the test set, the model adapts to the specific anomalies it will encounter, rather than relying on a generic definition of abnormality.

However, the model has specific requirements:

  • Contamination Rate ($\pi$): The model currently requires a prior estimate of the percentage of novel items in the test set.
  • Transductive Nature: Because the model must process the test set during training, it is not designed for real-time, streaming applications where data arrives one point at a time. It is best suited for batch processing scenarios.

5. Conclusion

TransductGAN represents a significant step forward in novelty detection by demonstrating the power of transductive learning and synthetic data generation. By transforming a one-class problem into a two-class problem, it provides a robust solution to the challenge of hyperparameter tuning and achieves superior detection accuracy on standard benchmarks. Future work may involve methods to automatically estimate the contamination rate, further automating the pipeline for real-world deployment.

Summary updated

Abstract

Novelty detection, a widely studied problem in machine learning, is the problem of detecting a novel class of data that has not been previously observed. A common setting for novelty detection is inductive whereby only examples of the negative class are available during training time. Transductive novelty detection on the other hand has only witnessed a recent surge in interest, it not only makes use of the negative class during training but also incorporates the (unlabeled) test set to detect novel examples. Several studies have emerged under the transductive setting umbrella that have demonstrated its advantage over its inductive counterpart. Depending on the assumptions about the data, these methods go by different names (e.g. transductive novelty detection, semi-supervised novelty detection, positive-unlabeled learning, out-of-distribution detection). With the use of generative adversarial networks (GAN), a segment of those studies have adopted a transductive setup in order to learn how to generate examples of the novel class. In this study, we propose TransductGAN, a transductive generative adversarial network that attempts to learn how to generate image examples from both the novel and negative classes by using a mixture of two Gaussians in the latent space. It achieves that by incorporating an adversarial autoencoder with a GAN network, the ability to generate examples of novel data points offers not only a visual representation of novelties, but also overcomes the hurdle faced by many inductive methods of how to tune the model hyperparameters at the decision rule level. Our model has shown superior performance over state-of-the-art inductive and transductive methods. Our study is fully reproducible with the code available publicly.
BibTeX
@misc{toron2022transductgan,
  author = {Najiba Toron and Janaina Mourão Miranda and John Shawe-Taylor},
  title = {{TransductGAN: a Transductive Adversarial Model for Novelty Detection}},
  year = {2022},
  eprint = {2203.15406},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.15406},
}
Figure