Abstract
Class-conditional generative models hold promise to overcome the shortcomings of their discriminative counterparts. They are a natural choice to solve discriminative tasks in a robust manner as they jointly optimize for predictive performance and accurate modeling of the input distribution. In this work, we investigate robust classification with likelihood-based generative models from a theoretical and practical perspective. Our analysis focuses on a spectrum of robustness properties: detection of worst-case outliers (adversarial examples), detection of average-case outliers (ambiguous inputs), and detection of incorrectly labeled in-distribution inputs. Our theoretical results reveal that it is impossible to guarantee detectability of adversarially-perturbed inputs even for near-optimal generative classifiers. Experimentally, we find that while we are able to train robust models for MNIST, robustness completely breaks down on CIFAR10. We relate this failure to various undesirable model properties that can be traced to the maximum likelihood training objective.
Introduction
Deep learning classifiers are notoriously fragile; a tiny, invisible "adversarial" perturbation to an image can flip a label from "panda" to "gibbon." A common hypothesis in the research community was that Conditional Generative Models (CGMs)—which learn the joint distribution of images and labels $p(x, y)$—would be more robust.
The logic was intuitive: because a CGM must model the entire input (every pixel), it shouldn't be able to ignore "class-irrelevant" information the way a standard discriminative model does. If an adversary pushes an image off the "data manifold," the generative model's likelihood should drop, signaling an outlier. This paper asks a critical question: Does this theoretical promise actually hold up in practice?
Theoretical Impossibility in High Dimensions
The authors provide a theoretical counter-example using high-dimensional geometry to prove that even a model with near-perfect density estimation can be forced into making confident, undetectable mistakes. In high-dimensional space, an adversary can move a point just a tiny distance $\Delta$ to flip the classification while actually increasing the likelihood under the model. This makes the attack mathematically undetectable by likelihood thresholds.
Experimental Methodology
The investigation utilizes Flow-based models (specifically GLOW) and PixelCNN++, which allow for the exact computation of log-likelihood. Three training configurations were tested:
- Base: A standard class-conditional Gaussian Mixture Model (GMM) prior.
- Reweighted: Adding an explicit classification loss to the generative objective to force better accuracy.
- Split Prior: Partitioning the latent space so that only a small subset of dimensions handles the class labels, preventing label information from being "washed out" by pixel noise.
Experiments were conducted on MNIST, CIFAR-10, and a new hybrid dataset, BG-MNIST, which consists of MNIST digits pasted onto complex CIFAR-10 backgrounds. This allows the researchers to keep the classification task easy while making the generative task hard.
Empirical Breakdown: MNIST vs. CIFAR-10
While the models showed some robustness on MNIST, they failed significantly on CIFAR-10. For example, using the "Boundary Attack," the models were fooled 100% of the time on CIFAR-10, and the resulting adversarial images were considered "highly likely" by the model.
A key insight is the Interpolation Failure. On MNIST, if you interpolate between two digits, the likelihood drops in the middle (as expected for "nonsense" images). However, on CIFAR-10, the opposite happens: interpolated, blurry images are often assigned higher likelihood than real images.
This failure is linked to the Maximum Likelihood Estimation (MLE) Dilemma. MLE is a "mode-covering" objective. In complex images, the "cost" of modeling the label is tiny compared to the cost of modeling thousands of pixels. Consequently, the model spends most of its capacity on backgrounds and textures rather than the features that define the class. The BG-MNIST experiment proved this: once you add a complex background to a simple digit, the generative model's robustness collapses, even though a human can still easily see the digit.
Conclusion
The standard Maximum Likelihood objective is fundamentally at odds with robust classification. While CGMs are the gold standard for density estimation, they prioritize "pixel-level" density over "semantic" density. Future work must look for new training objectives that prioritize semantic features if generative models are to truly deliver on their promise of robustness.
Summary updated