Abstract
Adversarial Training (AT) is known as an effective approach to enhance the robustness of deep neural networks. Recently, researchers have noticed that robust models trained with AT exhibit a surprising generative ability, synthesizing realistic images. However, the underlying reason for this phenomenon has remained under-explored. In this paper, we demystify this connection by developing a unified probabilistic framework called Contrastive Energy-based Models (CEM). We provide the first probabilistic characterization of AT through a unified understanding of robustness and generative ability. Furthermore, our framework extends to the unsupervised scenario, interpreting unsupervised contrastive learning as an importance sampling of CEM. Based on these insights, we propose principled methods for adversarial learning and sampling. Experiments demonstrate that our framework improves sample quality in both supervised and unsupervised learning. Notably, our unsupervised adversarial sampling method achieves an Inception Score (IS) of 9.61 on CIFAR-10, superior to previous energy-based models and comparable to state-of-the-art generative models.
Introduction: The Mystery of Robust Generativity
Adversarial Training (AT) is the gold standard for defending deep neural networks against malicious, imperceptible perturbations. Curiously, models trained this way develop a "generative" side effect: they learn features that align closely with human perception and can synthesize high-quality images.
This paper seeks to bridge the gap between adversarial robustness and generative modeling. By introducing CEM, we seek to explain why robust models are surprisingly good at generating realistic images and provide a principled way to sample from them, unifying supervised and unsupervised adversarial learning under a single probabilistic roof.
Methodology: Contrastive Energy-based Models (CEM)
The authors propose CEMs to model the joint distribution of two variables (such as an image and its label, or two different augmentations of the same image) based on their similarity.
The Theoretical Link
A key contribution of this work is the demonstration that the standard AT objective is mathematically equivalent to the gradient of the log-likelihood of a CEM. In this framework, the "inner maximization" (the adversarial attack) acts as a biased sampler, while the "outer minimization" (the model update) optimizes the energy landscape. This implies that adversarial attacks like PGD aren't just for finding vulnerabilities; they are effectively MCMC-like samplers that explore the model's energy surface.
Sampling Strategies
To leverage this generative potential, the paper introduces two novel sampling methods:
- Reinforced Conditional Sampling (RCS): For supervised models, RCS ensures generated images are both class-specific and realistic by injecting a "positive logsumexp gradient" to maintain high marginal probability.
- MaxEnt Sampling: For unsupervised models, MaxEnt updates random noise by maximizing the empirical entropy of the feature distribution, forcing the model to generate diverse and semantically meaningful samples.
Unifying InfoNCE and EBMs
The authors prove that the popular InfoNCE loss used in contrastive learning is actually an importance sampling estimate of a non-parametric CEM. This provides a theoretical foundation for unsupervised adversarial training, suggesting that standard contrastive learning is essentially a form of energy-based modeling.
Experimental Results
The researchers primarily used the CIFAR-10 dataset for evaluation, utilizing standard architectures including ResNet-18 and ResNet-50. The results indicate that CEM-based sampling methods produce high-quality results in fewer than 50 steps, significantly faster than the 1,000+ steps often required by Diffusion models.
As shown in the table above, the unsupervised CEM achieved an Inception Score (IS) of 9.61 on CIFAR-10. This is significantly higher than previous energy-based models like JEM (8.76) and is comparable to state-of-the-art GANs and Diffusion models. While the IS is excellent, the FID score (measuring how closely the generated distribution matches real data) still lags behind the very best Diffusion models, suggesting that while global structures are realistic, local textures can still contain artifacts.
Key Insights and Discussion
- Stability: AT is more stable than traditional EBM training because adversarial samples bridge the gap between real data and random noise more effectively than pure Langevin dynamics.
- Standard Training vs. CEM: The authors argue that standard (non-robust) training fails to be generative because it ignores the "negative phase" of energy-based modeling—it doesn't explore what the model shouldn't see as data, leading to a broken energy landscape.
- Robustness Performance: The proposed AT+CR (Consistency Regularization) objective, derived from the CEM framework, achieved 57.0% adversarial accuracy on WideResNet-34, matching the performance of the industry-standard TRADES method while maintaining better natural accuracy.
Conclusion
By framing Adversarial Training as a form of Contrastive Energy-based Modeling, this work provides a rigorous explanation for the generative capabilities of robust models. While challenges remain in scaling these models to high-resolution datasets like ImageNet, CEM offers a principled path toward unifying robustness and generation in deep learning.
Summary updated