WAIC, but Why? Generative Ensembles for Robust Anomaly Detection

Hyunsun Choi, Eric Jang, Alexander A. Alemi

Abstract

Machine learning models encounter Out-of-Distribution (OoD) errors when the data seen at test time are generated from a different stochastic generator than the one used to generate the training data. One proposal to scale OoD detection to high-dimensional data is to learn a tractable likelihood approximation of the training distribution, and use it to reject unlikely inputs. However, likelihood models on natural data are themselves susceptible to OoD errors, and even assign large likelihoods to samples from other datasets. To mitigate this problem, we propose Generative Ensembles, which robustify density-based OoD detection by way of estimating epistemic uncertainty of the likelihood model. We present a puzzling observation in need of an explanation -- although likelihood measures cannot account for the typical set of a distribution, and therefore should not be suitable on their own for OoD detection, WAIC performs surprisingly well in practice.

The Likelihood Paradox in High Dimensions

In an ideal scenario, a generative model trained on a specific dataset (e.g., CIFAR-10) should assign high likelihood to similar images and low likelihood to anomalous ones. However, recent findings reveal a counterintuitive behavior: high-dimensional generative models like VAEs and Flow-based models often assign higher likelihoods to out-of-distribution (OoD) data than to the data they were trained on.

Empirical distribution of negative bits per dimension (likelihood proxy) showing that OoD datasets like SVHN often receive higher likelihoods than the training set (CIFAR-10).

As shown in the figure above, a model trained on CIFAR-10 might perceive simpler datasets, such as SVHN digits, as more "likely." This occurs because the "typical set" where natural data resides is not necessarily the region of highest probability density in high-dimensional space. Models are often biased by low-level statistics; simpler images map to regions of the latent space with naturally higher density, leading to dangerous false positives in safety-critical applications.

Methods: Generative Ensembles and WAIC

To address this, we propose Generative Ensembles to estimate epistemic uncertainty—the degree to which models disagree on an input. By training an ensemble of $N$ independent models (e.g., VAEs or Glow) with different initializations, we can identify anomalies where the models' predictions diverge.

We utilize the Watanabe-Akaike Information Criterion (WAIC) as a robust detection metric: $$\text{WAIC}(x) = \mathbb{E}_\theta[\log p_\theta(x)] - \text{Var}_\theta[\log p_\theta(x)]$$

In this formulation, the first term represents the average log-likelihood, while the second term captures the variance across the ensemble. This variance acts as a penalty: if the models disagree significantly about an input, the WAIC score drops, effectively flagging the sample as an anomaly.

Key Insights: Uncertainty as a "Fence"

While a single model might erroneously assign a high likelihood to an OoD sample, it is highly improbable that multiple independently trained models will fail in the exact same way. The ensemble variance effectively "fences in" the true training distribution.

Distribution of the WAIC metric across different datasets. Note the significantly improved separation between the training set (CIFAR-10) and OoD sets compared to raw likelihood.

Our results demonstrate that WAIC successfully overcomes the typicality paradox. On the challenging CIFAR-10 vs. SVHN task, where a single Glow model fails (AUROC of 7.5), the WAIC ensemble achieves a perfect 100.0 AUROC. This suggests that while likelihood alone is insufficient, the sensitivity of WAIC to different "annuli" (rings) in latent space allows it to distinguish between datasets with high precision.

Results and Real-World Application

Beyond standard image benchmarks, Generative Ensembles show remarkable performance in practical scenarios:

  • Unsupervised Superiority: WAIC consistently outperformed popular supervised methods like ODIN, despite requiring no labeled OoD data during training.
  • Credit Fraud Detection: On the Kaggle Credit Fraud dataset, WAIC achieved an AUROC of 94.7, rivaling supervised classifiers (99.1) that had the advantage of seeing fraudulent examples beforehand.

Conclusion

Generative Ensembles provide a powerful, unsupervised framework for robust anomaly detection. While the theoretical reason for WAIC's success in high dimensions remains an area of active research, its ability to penalize high-likelihood OoD samples with epistemic uncertainty offers a significant step toward safer and more reliable machine learning systems.

Summary updated

Abstract

Machine learning models encounter Out-of-Distribution (OoD) errors when the data seen at test time are generated from a different stochastic generator than the one used to generate the training data. One proposal to scale OoD detection to high-dimensional data is to learn a tractable likelihood approximation of the training distribution, and use it to reject unlikely inputs. However, likelihood models on natural data are themselves susceptible to OoD errors, and even assign large likelihoods to samples from other datasets. To mitigate this problem, we propose Generative Ensembles, which robustify density-based OoD detection by way of estimating epistemic uncertainty of the likelihood model. We present a puzzling observation in need of an explanation -- although likelihood measures cannot account for the typical set of a distribution, and therefore should not be suitable on their own for OoD detection, WAIC performs surprisingly well in practice.
BibTeX
@misc{choi2018waic,
  author = {Hyunsun Choi and Eric Jang and Alexander A. Alemi},
  title = {{WAIC, but Why? Generative Ensembles for Robust Anomaly Detection}},
  year = {2018},
  eprint = {1810.01392},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.01392},
}
Figure