Why Normalizing Flows Fail to Detect Out-of-Distribution Data

Polina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson

Abstract

Detecting out-of-distribution (OOD) data is crucial for robust machine learning systems. Normalizing flows are flexible deep generative models that often surprisingly fail to distinguish between in- and out-of-distribution data: a flow trained on pictures of clothing assigns higher likelihood to handwritten digits. We investigate why normalizing flows perform poorly for OOD detection. We demonstrate that flows learn local pixel correlations and generic image-to-latent-space transformations which are not specific to the target image dataset. We show that by modifying the architecture of flow coupling layers we can bias the flow towards learning the semantic structure of the target data, improving OOD detection. Our investigation reveals that properties that enable flows to generate high-fidelity images can have a detrimental effect on OOD detection.

The Puzzling Failure of Likelihood

In a robust machine learning system, a generative model trained on a specific dataset—such as faces (CelebA)—should ideally assign high likelihood to similar new data and low likelihood to unrelated data, like house numbers (SVHN). However, normalizing flows frequently exhibit a counter-intuitive behavior: they assign higher likelihoods to "simpler" datasets they have never encountered during training.

Log-likelihood distributions and internal network visualizations for in-distribution (ImageNet) and OOD (CelebA) data.

This phenomenon suggests that the maximum likelihood objective is agnostic to OOD detection. Instead, the model's "internal compass" is driven by its inductive biases—the architectural assumptions that dictate what the model considers "likely."

Forensic Analysis: Graphical vs. Semantic Biases

To understand this failure, we conducted a forensic investigation into popular flow architectures like RealNVP and Glow. Our analysis revealed several key insights:

  • Graphical Dominance: Flows are biased toward learning "graphical" properties, such as local pixel correlations (e.g., neighboring pixels having similar colors), rather than high-level "semantic" properties (e.g., the structural components of a face).
  • Leaky Latent Spaces: Visualizations of latent representations ($z$) show that they often retain the edges and shapes of input images. Rather than abstracting the data into semantic concepts, the mapping primarily rearranges raw pixel information.
  • Coupling Layer Co-adaptation: A critical finding is that subsequent coupling layers "cheat" by encoding information about masked pixels into unmasked regions. This co-adaptation allows the model to predict pixels with high certainty, inflating likelihood scores without achieving a global understanding of the image structure.
  • Background Influence: Likelihood calculations are often dominated by background statistics. Large patches of uniform color make simpler datasets appear more "likely" than complex ones, regardless of the training distribution.

Improving OOD Detection

We explored several strategies to mitigate these biases and force the model to focus on semantic content:

  1. Architectural Bottlenecks: By introducing dimensionality bottlenecks in the internal scale ($s$) and shift ($t$) networks, we forced the model to ignore local pixel noise. This significantly improved OOD detection for pairs like FashionMNIST vs. MNIST.
  2. Semantic Embeddings: The most effective solution involved training flows on high-level semantic embeddings from a pre-trained EfficientNet rather than raw pixels. This approach bypassed low-level graphical biases, achieving nearly 100% AUROC for SVHN and CelebA detection.
  3. Domain Specificity: Interestingly, normalizing flows performed much better at OOD detection on tabular data (e.g., UCI datasets), where local pixel-like correlations are absent. This confirms that the failure is largely an image-specific architectural bias.

Conclusion

Our investigation reveals that the very properties enabling flows to generate high-fidelity images—such as capturing fine-grained local correlations—can be detrimental to OOD detection. While architectural tweaks like bottlenecks and cycle-masks offer improvements, the most robust results come from leveraging pre-trained semantic features. Future research should focus on developing flow architectures that naturally learn hierarchical, semantic representations without relying on external supervision.

Summary updated

Abstract

Detecting out-of-distribution (OOD) data is crucial for robust machine learning systems. Normalizing flows are flexible deep generative models that often surprisingly fail to distinguish between in- and out-of-distribution data: a flow trained on pictures of clothing assigns higher likelihood to handwritten digits. We investigate why normalizing flows perform poorly for OOD detection. We demonstrate that flows learn local pixel correlations and generic image-to-latent-space transformations which are not specific to the target image dataset. We show that by modifying the architecture of flow coupling layers we can bias the flow towards learning the semantic structure of the target data, improving OOD detection. Our investigation reveals that properties that enable flows to generate high-fidelity images can have a detrimental effect on OOD detection.
BibTeX
@inproceedings{kirichenko2020why,
  author = {Polina Kirichenko and Pavel Izmailov and Andrew Gordon Wilson},
  title = {{Why Normalizing Flows Fail to Detect Out-of-Distribution Data}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=a2Y37GZqUs},
}
Figure