Introduction
Deep generative models, particularly Normalizing Flows, are powerful tools for density estimation. However, they frequently suffer from "likelihood flips," where out-of-distribution (OOD) data is assigned higher probability than in-distribution (ID) training data. This phenomenon undermines their reliability in safety-critical applications. This paper introduces HYBOOD, a hybrid model designed to detect OOD data while simultaneously quantifying the impact of various data corruptions through a novel ranking framework.
The challenge in OOD detection is twofold: distinguishing between covariate shift (degraded image quality of known classes) and semantic shift (entirely new classes), and understanding how specific corruptions affect model performance.
Methodology: The HYBOOD Architecture
HYBOOD leverages a GLOW-based normalizing flow architecture integrated with a simplified discriminative component. Unlike traditional hybrid models that use complex Generalized Linear Models (GLMs) in the latent space, HYBOOD employs Global Average Pooling and a Linear Model (GAPLM). This modification drastically reduces the parameter count—from approximately 15,000 to just 240 in experimental settings—without sacrificing classification accuracy.
The model computes the joint distribution $p(x, y)$ in a single feed-forward pass. By combining generative density $p(x)$ with predictive confidence $p(y|x)$, HYBOOD creates a robust framework for OOD detection that accounts for both data structure and label consistency.
Corruption Difficulty Ranking (CDR)
A primary contribution of this work is the Corruption Difficulty Ranking (CDR) metric. CDR provides a diagnostic tool to measure how much a specific corruption (e.g., fog, noise, or blur) degrades model performance. It synthesizes three distinct measures:
- Wasserstein Distance (WD): Quantifies the shift of the corrupted distribution from the ID distribution within the latent space.
- Mutual Information (MI): Evaluates the dependency between test inputs and the training distribution.
- Minimum Description Length (MDL): Serves as an agnostic measure of image complexity, reflecting the difficulty of compression.
Experimental Results and Insights
The authors evaluated HYBOOD on MNIST and CIFAR-10, along with their corrupted counterparts (MNIST-C, CIFAR10-C) and semantic OOD sets (Fashion-MNIST, SVHN).
The "Sandwich" Effect and Separability
Analysis of log-density distributions revealed a "sandwich" effect. For minor corruptions (e.g., simple scaling), ID data often resides in the center of the log-density range. As corruption severity increases (e.g., fog or impulse noise), the corrupted data moves further away from the ID distribution, becoming more distinguishable but also more detrimental to classification accuracy.
CDR Validation
The CDR metric demonstrates a strong correlation with model degradation. Corruptions ranked as highly difficult by CDR consistently align with the lowest F1 scores in classification tasks.
Key results include:
- Efficiency: The GAPLM head achieved a 98% reduction in parameters with negligible performance loss (e.g., 0.9740 vs 0.9769 F1 on MNIST).
- Robustness: HYBOOD effectively separates ID, covariate shift, and semantic shift based on their respective log-density intervals, providing a clearer boundary for OOD detection.
Conclusion
HYBOOD addresses the likelihood flip problem in generative models by combining density estimation with discriminative confidence. The introduction of the CDR metric provides a valuable diagnostic framework for understanding model sensitivity to environmental noise and corruptions. Future work will focus on automating hyperparameter selection and extending the framework to non-image modalities.
Summary updated