One-Class Learned Encoder-Decoder Network With Adversarial Context Masking for Novelty Detection

John Taylor Jewell, Vahid Reza Khazaie, Yalda MohsenzadehWACV 2022

TL;DR

This paper introduces OLED, a novel one-class novelty detection method that uses an adversarial approach to learn optimal image masking. By adversarially training a "Mask Module" to maximize reconstruction error and a "Reconstructor" to minimize it, OLED effectively identifies important image regions for anomaly detection, leading to superior performance on benchmark datasets.

What Problem Does This Paper Address?

Novelty detection, also known as anomaly detection, is the task of identifying samples that deviate from a known distribution of "normal" data. This is a challenging problem because, by definition, anomalous samples are rare and often unknown during training, making traditional supervised classification methods unsuitable. Deep autoencoders (AEs) have emerged as a popular tool for this task, as they can learn to reconstruct normal data well, leading to higher reconstruction errors for anomalies.

However, existing methods like Context Autoencoders (CAEs), which use random masking to force the AE to learn more robust representations, suffer from a key limitation: random masking doesn't consistently target the most informative parts of an image. This can lead to suboptimal representations and inconsistent anomaly scores, especially when anomalies share some visual similarities with normal data. The paper aims to overcome this by learning where to mask an image to maximize the difficulty for the autoencoder to reconstruct it, thereby improving its ability to distinguish between normal and anomalous samples.

Methods

The core of the proposed method, OLED (One-Class Learned Encoder-Decoder Network with Adversarial Context Masking), lies in its adversarial training framework. It consists of two main components:

  • Reconstructor (R): A convolutional encoder-decoder network trained to reconstruct masked input images. It aims to minimize the reconstruction error.
  • Mask Module (MM): This module, itself composed of a Mask Generator (MG) and a Threshold Unit (T), learns to generate masks that highlight important image regions. The MM's objective is to maximize the reconstruction error produced by R.

The adversarial training works as follows: MM generates a mask, which is then applied to the input image to create a masked image. R attempts to reconstruct this masked image. MM is trained to produce masks that make this reconstruction difficult (i.e., result in a high reconstruction error), while R is trained to get better at reconstructing the masked images (i.e., minimize the reconstruction error). This creates a "minimax game" where MM learns to identify the most discriminative parts of the image for anomaly detection.

The MM generates masks by first creating an activation map (via MG) and then applying a threshold (via T) to produce a binary mask. This learned masking is applied to the input image before it's fed to the Reconstructor. The paper also introduces three distinct anomaly scores derived from different loss terms: Srec (global reconstruction), Smask (masked region reconstruction), and Scont (contextual reconstruction), which can be combined for a more robust anomaly score (Savg).

Datasets

The paper evaluates OLED on three standard datasets for novelty detection:

  • MNIST: A dataset of handwritten digits (0-9), used for grayscale image anomaly detection.
  • CIFAR-10: A dataset of natural images across ten object classes, used for RGB image anomaly detection. This dataset is noted as being more challenging due to inter-class variations and unaligned backgrounds.
  • UCSD: A dataset of pedestrian and outdoor scenes, used for video anomaly detection. The paper specifically uses the Ped2 subset.

Key Insights and Ideas

  • Adversarial Masking is Key: The most significant insight is that learning where to mask an image is crucial for effective novelty detection, outperforming random masking. By forcing the network to reconstruct challenging regions, it learns more discriminative features.
  • Masking Improves Robustness: OLED's learned masking helps prevent anomalies from being reconstructed too well, a common pitfall in traditional autoencoder-based anomaly detection.
  • Multiple Anomaly Scores: The framework naturally yields multiple anomaly scores (Srec, Smask, Scont) that capture different aspects of normality, offering a more nuanced detection capability.
  • End-to-End Training: The entire OLED network is trained end-to-end, simplifying the training pipeline compared to methods that rely on pre-trained components or complex multi-stage processes.
  • Generalizability of Masking: The Mask Module shows promise for other computer vision tasks beyond anomaly detection, as demonstrated by its strong performance in semantic segmentation.

Key Results

OLED demonstrates state-of-the-art performance across all evaluated datasets:

  • MNIST: Achieved an average AUCROC of 0.9845 for the combined anomaly score (Savg), surpassing methods like MemAE (0.9751).
  • CIFAR-10: Achieved an AUCROC of 0.6683 for Savg, outperforming OCGAN (0.6566) and other recent methods.
  • UCSD (Video): Achieved a frame-level AUCROC of 99.02% and an Equal Error Rate (EER) of 5.4%, significantly outperforming existing methods.
  • Mask Module Effectiveness: The Mask Module itself showed strong performance in segmenting important image regions, achieving an AUCROC of 0.8499 for inliers and 0.8472 for outliers on a segmentation task derived from MNIST.

Limitations and Future Work

  • Hyperparameter Tuning: The paper mentions that the threshold parameter for masking can be adjusted based on dataset difficulty, implying that optimal settings might require tuning.
  • Computational Cost: While not explicitly detailed as a limitation, adversarial training can be computationally intensive.
  • Future Work: The authors suggest that the learned masking approach has the potential to be applied to other modalities of data and broader computer vision tasks beyond anomaly detection. They also highlight that the MM can be seamlessly integrated into existing AE-based anomaly detection methods.

Summary updated

BibTeX
@inproceedings{jewell2022one,
  author = {John Taylor Jewell and Vahid Reza Khazaie and Yalda Mohsenzadeh},
  title = {{One-Class Learned Encoder-Decoder Network With Adversarial Context Masking for Novelty Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure