Introduction
Adversarial Training (AT) is widely recognized as the gold standard for enhancing the robustness of deep neural networks against malicious perturbations. Recently, an intriguing phenomenon has emerged: robust models trained via AT possess a latent "generative" capability, allowing them to synthesize realistic images when inverted.
This paper introduces Contrastive Energy-based Models (CEM), a unified probabilistic framework that mathematically explains why AT makes models both robust and capable of generating high-quality images. By bridging the gap between supervised robustness and unsupervised contrastive learning, the authors develop novel sampling methods that allow robust models to synthesize images with state-of-the-art efficiency.
The Problem: Demystifying the Generative Soul of AT
While researchers have observed that robust classifiers can be inverted to generate realistic images, two major questions have remained under-explored:
- The "Why": Why does a training process explicitly designed for defensive classification accidentally yield a generative model?
- The "Unsupervised Gap": AT typically requires labeled data. Can we achieve the same robustness and generative quality in an unsupervised setting (without labels)?
This paper tackles these questions by demystifying the probabilistic mechanics of AT and extending them to unsupervised learning scenarios.
Methodology: Contrastive Energy-based Models (CEM)
The authors propose the Contrastive Energy-based Model (CEM) framework. In a standard Energy-Based Model (EBM), the probability of a data point is inversely proportional to an "energy" function. The CEM defines this energy based on the similarity between two inputs.
- Parametric CEM (P-CEM): Utilized for supervised learning, where the model learns the similarity between an image and its corresponding class label.
- Non-Parametric CEM (NP-CEM): Utilized for unsupervised learning, where the model learns the similarity between two different augmentations of the same image.
Key Theoretical Insights
The core technical contribution is demonstrating that Adversarial Training is mathematically equivalent to Maximum Likelihood Training of a CEM.
- AT as Generative Modeling: The minimax objective of AT is not merely a defensive mechanism; it trains a joint probability distribution of data and labels. The "inner maximization" (the adversarial attack) acts as a biased form of sampling from the model's distribution, while the "outer minimization" (the model update) serves as the learning step.
- PGD as a Sampler: Projected Gradient Descent (PGD), typically used to craft adversarial examples, is shown to be a competitive, faster, and more stable alternative to Langevin dynamics (a standard EBM sampling method) during training.
- The InfoNCE Connection: The authors reveal that InfoNCE, the prevalent loss function for unsupervised contrastive learning, is actually an "importance sampling" approximation of their NP-CEM. This provides a long-sought probabilistic foundation for why contrastive learning works.
Novel Sampling Algorithms
Based on this framework, the authors introduce two new sampling algorithms requiring fewer than 50 steps (vastly more efficient than the 1,000+ steps often needed by score-based or diffusion models):
- Reinforced Conditional Sampling (RCS): For supervised models, this improves image quality by injecting a "positive" gradient that pushes samples toward the data manifold.
- MaxEnt (Maximum Entropy) Sampling: A novel approach to generate images from unsupervised models by maximizing the entropy of a batch of samples in the feature space.
Key Results
The paper demonstrates that robust models can compete with dedicated generative architectures on the CIFAR-10 dataset.
As detailed in the table above, the CEM framework achieves impressive generative metrics:
- Supervised Generation: The supervised CEM on ResNet-50 achieves an Inception Score (IS) of 9.80. This is on par with dedicated generators like StyleGAN2-ADA (10.02) and significantly outperforms previous AT-based generators like TA (7.5).
- Unsupervised Generation: The unsupervised CEM reaches an IS of 9.61, outperforming previous energy-based models like JEM (8.76).
- Robustness: Beyond generation, the authors introduced a new regularization term (AT+CR) derived from their framework. It achieves 57.0% adversarial accuracy on WideResNet-34, matching state-of-the-art methods like TRADES while maintaining superior natural accuracy.
Limitations and Future Work
While the Inception Scores (IS) are highly competitive, indicating excellent semantic clarity and global structure, the Fréchet Inception Distance (FID) scores for CEM models (e.g., 40.25 - 55.91) trail behind those of diffusion and score-based models (e.g., NCSN++ at 2.20). This mismatch suggests that while the models understand the global data distribution, there are still local artifacts or textural differences compared to real data.
Future work will likely focus on refining these local textures to improve FID and scaling the CEM framework to higher-resolution datasets like ImageNet, where EBMs traditionally face stability challenges.
Summary updated