Abstract
Class-conditional generative models hold promise to overcome the shortcomings of their discriminative counterparts. They are a natural choice to solve discriminative tasks in a robust manner as they jointly optimize for predictive performance and accurate modeling of the input distribution. In this work, we investigate robust classification with likelihood-based generative models from a theoretical and practical perspective to investigate if they can deliver on their promises. Our analysis focuses on a spectrum of robustness properties: (1) Detection of worst-case outliers in the form of adversarial examples; (2) Detection of average-case outliers in the form of ambiguous inputs and (3) Detection of incorrectly labeled in-distribution inputs. Our theoretical result reveals that it is impossible to guarantee detectability of adversarially-perturbed inputs even for near-optimal generative classifiers. Experimentally, we find that while we are able to train robust models for MNIST, robustness completely breaks down on CIFAR10. We relate this failure to various undesirable model properties that can be traced to the maximum likelihood training objective. Despite being a common choice in the literature, our results indicate that likelihood-based conditional generative models may are surprisingly ineffective for robust classification.
The Promise of Generative Robustness
Deep learning classifiers are notoriously brittle; small, adversarial perturbations can flip a model's prediction while remaining invisible to humans. A popular proposed solution is the use of Conditional Generative Models (CGMs). Unlike standard classifiers that only learn the decision boundary ($p(y|x)$), CGMs learn the joint distribution of data and labels ($p(x, y)$). The intuition is that a CGM should be harder to fool because it "knows" what a valid input looks like. If an attacker moves an image off the data manifold to cause a misclassification, the model should theoretically detect a drop in likelihood and flag the input as an outlier.
Theoretical Impossibility
Despite the intuitive appeal, the authors provide a rigorous mathematical proof showing that near-optimal likelihood and high accuracy do not guarantee robustness. Using a high-dimensional "annulus" construction, they demonstrate that in high-dimensional space, the volume of "confident mistake" regions can be massive even if the probability mass is small. This means an adversary can almost always find a path to a wrong class without leaving the high-likelihood region of the model.
Empirical Breakdown: MNIST vs. CIFAR10
The researchers evaluated Flow-based generative models (specifically GLOW) across three dimensions: worst-case outliers (adversarial examples), average-case outliers (ambiguous inputs), and mislabeled data.
While the models performed well on simple datasets like MNIST—achieving ~99% accuracy and high resistance to attacks—robustness completely collapsed on CIFAR10. On complex data, the "Boundary Attack" achieved a 100% success rate in finding adversarial examples that the model considered high-likelihood.
A key insight into this failure is the Interpolation Failure. On MNIST, if you interpolate between a "7" and a "0", the likelihood drops in the middle as the model recognizes the blurred image as an outlier. On CIFAR10, the opposite occurs: interpolated images often have higher likelihood than real images, suggesting the model "mashes" distributions together rather than keeping them distinct.
The Likelihood Trap
Why do these models fail on complex data? The authors point to the Maximum Likelihood objective. MLE treats every bit of information in an image as equally important. In natural images, the bits required to describe background textures and lighting vastly outnumber those required to identify the object (e.g., "cat-ness"). Consequently, the model spends most of its capacity modeling background noise, making it insensitive to class-specific changes.
This was confirmed using a hybrid dataset, BG-MNIST, where MNIST digits were placed on CIFAR10 backgrounds. As the entropy and complexity of the background increased, the model's robustness vanished, proving that the failure on CIFAR10 is driven by background information overwhelming the generative objective.
Conclusion
The study concludes that likelihood-based CGMs are surprisingly ineffective for robust classification in the real world. For generative models to deliver on their promises, new training objectives are needed that can prioritize semantic, class-relevant features over low-level pixel noise without losing the benefits of joint distribution modeling.
Summary updated