Where's Wally Now? Deep Generative and Discriminative Embeddings for Novelty Detection

Philippe Burlina, Neil Joshi, I-Jeng WangCVPR 2019 · DOI

Novelty Detection (ND)—the ability of a machine learning model to identify data that deviates from its training distribution—is a cornerstone of robust AI. It is essential for applications ranging from autonomous vehicles identifying unknown obstacles to medical systems flagging rare pathologies. However, the field currently lacks standardization, often relying on low-resolution datasets and failing to account for the inherent difficulty of specific detection tasks. This paper introduces a unified framework to evaluate deep embeddings and proposes a new complexity metric to bring rigor to ND research.

The Problem: A "Confusing State of Affairs"

Most existing ND methods are benchmarked on simplistic datasets like MNIST or CIFAR-10. While useful for initial testing, these do not reflect the complexity of "in-the-wild" scenarios. Furthermore, there is no standard way to measure the difficulty of a novelty detection task. Without a metric for problem complexity, comparing two algorithms is often an "apples-to-oranges" comparison; a model may appear superior simply because it was tested on an easier split of inliers and outliers.

A Unified Novelty Detection Framework

The authors propose a two-step framework designed to evaluate both discriminative and generative approaches to feature extraction:

  1. Embedding: Input images are mapped to a feature vector (embedding). The study compares discriminative networks (VGG16 pre-trained on ImageNet) against generative networks (a GAN-based architecture known as Ganomaly).
  2. Scoring: These embeddings are processed by a novelty scorer to determine outlier status. The researchers tested four classic algorithms: One-Class SVM (1CSVM), Local Outlier Factor (LOF), Elliptic Envelope (EE), and Isolation Forest (IF).
The proposed novelty detection framework, illustrating the parallel paths for discriminative and generative embeddings followed by a novelty scoring stage.

Measuring Task Difficulty with BERCA

A major contribution of this work is the Bayes Error Rate Complexity Assessment (BERCA). This metric uses a K-Nearest Neighbor approach to estimate the theoretical minimum error rate for any given inlier/outlier split. By quantifying task complexity, researchers can plot performance (ROC-AUC) against difficulty, providing a clearer view of a model's robustness as the task becomes more challenging.

New Benchmarks and Protocols

To move beyond low-resolution baselines, the authors introduced IN-125, a benchmark consisting of 125 ImageNet classes that were not part of the original 1,000-class challenge. This prevents pre-trained models from having an unfair advantage. The study also standardizes two protocols:

  • SIMO (Single Inlier, Multiple Outliers): Learning one class and identifying everything else as novel.
  • MISO (Multiple Inliers, Single Outlier): Learning a broad set of "normal" data and identifying a specific novel class.
Qualitative examples showing the model's ability to distinguish between predicted and actual inliers/outliers across different animal and object categories.

Key Insights and Results

The research yielded several critical findings for the ND community:

  • Discriminative Dominance: Discriminative embeddings generally outperform generative ones, largely due to the "prior knowledge" gained during pre-training. The best discriminative setup (LOF/$X_{GAP512}$) achieved an AUC of 0.6224 on CIFAR-10 (MISO).
  • Generative Progress: The authors established a new state-of-the-art for generative methods by applying a 1CSVM to the latent vector ($X_{GAN}$) of a Ganomaly encoder. This setup reached an AUC of 0.5792 on IN-125, outperforming traditional reconstruction-error-based methods.
  • Complexity Correlation: Performance consistently degrades as BERCA complexity increases, validating the metric as a reliable tool for benchmarking.

Future Directions

While generative methods are improving, they still lag behind discriminative models. Future work will likely explore more sophisticated generative architectures like BigGAN or StyleGAN to create richer latent spaces. Additionally, the authors suggest incorporating "semantic complexity"—measuring the conceptual distance between classes using hierarchies like WordNet—to further refine how we evaluate novelty detection in complex environments.

Summary updated

BibTeX
@inproceedings{burlina2019where,
  author = {Philippe Burlina and Neil Joshi and I-Jeng Wang},
  title = {{Where's Wally Now? Deep Generative and Discriminative Embeddings for Novelty Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  doi = {10.1109/cvpr.2019.01177},
}
Figure