New Perspective on Progressive GANs Distillation for One-class Novelty Detection

Zhiwei Zhang, Dong Yu, Hanyu Peng, Shifeng Chen

TL;DR

This paper introduces a novel approach for one-class novelty detection by leveraging Generative Adversarial Networks (GANs) and knowledge distillation. They propose an EDE-GAN architecture that uses latent space distances for anomaly scoring and demonstrate that a batch size of 1 is optimal for this task. To create efficient models, they develop Progressive Knowledge Distillation with GANs (P-KDGAN), which transfers knowledge from a larger teacher GAN to a smaller student GAN through a two-step learning process, achieving state-of-the-art performance with significant compression.

What Problem Does This Paper Address?

One-class novelty detection (ND) aims to identify data points that deviate from a known distribution of "normal" data. This is crucial in applications like fraud detection, network intrusion detection, and medical diagnosis. Traditional methods often struggle with high-dimensional data. While GANs have shown promise, existing GAN-based ND methods often rely on reconstruction errors, which can be sensitive to noise. Furthermore, deploying these powerful deep learning models on resource-constrained devices is challenging due to their computational and storage demands. This paper addresses these issues by proposing a more robust GAN-based anomaly scoring mechanism and a knowledge distillation technique to create efficient, high-performing models for one-class novelty detection.

Methods

The core of this work lies in two main contributions:

  1. EDE-GAN for Novelty Detection: The authors propose an Encoder-Decoder-Encoder (EDE) GAN architecture. Unlike previous methods that used reconstruction error between input and output images as an anomaly score, their EDE-GAN calculates the anomaly score based on the distance between two latent vectors. They also investigate the impact of batch size, finding that a batch size of 1 yields the best performance, which is counter-intuitive for standard classification tasks but attributed to better model fitting and generalization in this specific ND context. They also highlight the benefit of constraining the latent space during training.
  2. Progressive Knowledge Distillation with GANs (P-KDGAN): To address the computational burden of large GANs, they introduce P-KDGAN. This method employs a teacher-student paradigm where a pre-trained, high-performing teacher GAN transfers its knowledge to a smaller, more efficient student GAN. The key innovation is a two-step progressive learning process:
  3. Step 1 (Basic Knowledge Learning): The student GAN is trained using a novel distillation loss that measures similarities between intermediate representations (latent vectors and reconstructed images) of the teacher and student. This step focuses on learning fundamental patterns from the teacher.
  4. Step 2 (Fine-Learning): The student GAN is further trained to refine its performance, potentially even surpassing the teacher in some aspects. This progressive approach aims to improve both accuracy and stability. They define four specific distillation structures (KDGAN-① to KDGAN-④) based on different combinations of teacher and student losses and whether the teacher is fixed or updated.

Datasets

The proposed methods were evaluated on three standard benchmark datasets:

  • CIFAR-10: A dataset of 60,000 32x32 RGB images across 10 classes.
  • MNIST: A dataset of 70,000 28x28 grayscale handwritten digit images.
  • FMNIST (Fashion-MNIST): A dataset of 70,000 28x28 grayscale images of fashion products from 10 categories.

For each dataset, one class was designated as "normal" for training, and the remaining classes were used as "novelty" or "outlier" samples during testing.

Key Insights and Ideas

  • Batch Size 1 is Optimal for EDE-GAN: This is a significant finding for novelty detection. The authors attribute this to the specific nature of the task, where learning individual normal samples well and using latent space constraints helps prevent overfitting, which is exacerbated by larger batch sizes in standard GAN training.
  • Latent Space Constraint is Crucial: The paper demonstrates that constraining the latent space during training, as done in their proposed EDÊ-GAN (a variant of EDE-GAN), leads to improved performance and a more robust anomaly score.
  • Progressive Distillation Outperforms Single-Step: The two-step progressive learning in P-KDGAN is shown to be more effective than single-step distillation, leading to better performance and stability in the student GAN.
  • Student GANs Can Surpass Teacher GANs: In some cases, the fine-tuned student GANs trained with P-KDGAN achieved better performance than the original teacher GAN, showcasing the power of the distillation process.
  • Novelty Detection Requires Rethinking Batch Normalization: The authors suggest that the standard Batch Normalization (BN) might need re-evaluation for novelty detection tasks, as its behavior with small batch sizes and its reliance on batch statistics might not be ideal.

Key Results

  • EDE-GAN Achieves State-of-the-Art: The EDE-GAN model, particularly with a batch size of 1, achieved state-of-the-art performance on the tested datasets.
  • Significant Compression with P-KDGAN: P-KDGAN achieved substantial compression ratios:
  • CIFAR-10: 24.45:1 compression, with only a 0.71% drop in AUC compared to the teacher.
  • MNIST: 311.11:1 compression, with a 0.55% drop in AUC.
  • FMNIST: 700:1 compression, with a 0.18% drop in AUC.
  • Improved AUC: P-KDGAN improved the student GAN's performance by 2.44% on CIFAR-10, 1.77% on MNIST, and 1.73% on FMNIST compared to training the student from scratch.
  • Competitive Performance: The proposed method (EDE-GAN) achieved competitive or superior results compared to several traditional and deep learning-based novelty detection methods on CIFAR-10 and MNIST. For instance, on CIFAR-10, it achieved 73.76% AUC, outperforming OCGAN by about 8%.

Limitations and Future Work

  • Batch Normalization Re-evaluation: The authors explicitly state that novelty detection might require rethinking how Batch Normalization is improved or adapted for this specific task, given its sensitivity to batch statistics.
  • Generalization of P-KDGAN: While P-KDGAN shows promise for compression, its application to other tasks like image dehazing, super-resolution, and image synthesis is suggested as future work.
  • Batch Size Impact: The paper highlights the importance of considering the "last epoch" performance rather than just the "best epoch" when evaluating novelty detection models, especially concerning batch size variations.
  • Exploration of Distillation Structures: While four distillation structures were proposed, further exploration into optimal combinations and their impact on different datasets could be beneficial.

Summary updated

Abstract

One-class novelty detection is conducted to identify anomalous instances, with different distributions from the expected normal instances. In this paper, the Generative Adversarial Network based on the Encoder-Decoder-Encoder scheme (EDE-GAN) achieves state-of-the-art performance. The two factors bellow serve the above purpose: 1) The EDE-GAN calculates the distance between two latent vectors as the anomaly score, which is unlike the previous methods by utilizing the reconstruction error between images. 2) The model obtains best results when the batch size is set to 1. To illustrate their superiority, we design a new GAN architecture, and compare performances according to different batch sizes. Moreover, with experimentation leads to discovery, our result implies there is also evidence of just how beneficial constraint on the latent space are when engaging in model training. In an attempt to learn compact and fast models, we present a new technology, Progressive Knowledge Distillation with GANs (P-KDGAN), which connects two standard GANs through the designed distillation loss. Two-step progressive learning continuously augments the performance of student GANs with improved results over single-step approach. Our experimental results on CIFAR-10, MNIST, and FMNIST datasets illustrate that P-KDGAN improves the performance of the student GAN by 2.44%, 1.77%, and 1.73% when compressing the computationat ratios of 24.45:1, 311.11:1, and 700:1, respectively.
BibTeX
@misc{zhang2021new,
  author = {Zhiwei Zhang and Dong Yu and Hanyu Peng and Shifeng Chen},
  title = {{New Perspective on Progressive GANs Distillation for One-class Novelty Detection}},
  year = {2021},
  eprint = {2109.07295},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.07295},
}
Figure