A Unified Contrastive Energy-based Model for Understanding the Generative Ability of Adversarial Training

Yifei Wang, Yisen Wang, Jiansheng Yang, Zhouchen Lin

Introduction

Adversarial Training (AT) is widely recognized as the gold standard for enhancing the robustness of deep neural networks against malicious perturbations. Recently, an intriguing phenomenon has emerged: robust models trained via AT possess a latent "generative" capability, allowing them to synthesize realistic images when inverted.

This paper introduces Contrastive Energy-based Models (CEM), a unified probabilistic framework that mathematically explains why AT makes models both robust and capable of generating high-quality images. By bridging the gap between supervised robustness and unsupervised contrastive learning, the authors develop novel sampling methods that allow robust models to synthesize images with state-of-the-art efficiency.

The Problem: Demystifying the Generative Soul of AT

While researchers have observed that robust classifiers can be inverted to generate realistic images, two major questions have remained under-explored:

  1. The "Why": Why does a training process explicitly designed for defensive classification accidentally yield a generative model?
  2. The "Unsupervised Gap": AT typically requires labeled data. Can we achieve the same robustness and generative quality in an unsupervised setting (without labels)?

This paper tackles these questions by demystifying the probabilistic mechanics of AT and extending them to unsupervised learning scenarios.

Methodology: Contrastive Energy-based Models (CEM)

The authors propose the Contrastive Energy-based Model (CEM) framework. In a standard Energy-Based Model (EBM), the probability of a data point is inversely proportional to an "energy" function. The CEM defines this energy based on the similarity between two inputs.

  • Parametric CEM (P-CEM): Utilized for supervised learning, where the model learns the similarity between an image and its corresponding class label.
  • Non-Parametric CEM (NP-CEM): Utilized for unsupervised learning, where the model learns the similarity between two different augmentations of the same image.

Key Theoretical Insights

The core technical contribution is demonstrating that Adversarial Training is mathematically equivalent to Maximum Likelihood Training of a CEM.

  • AT as Generative Modeling: The minimax objective of AT is not merely a defensive mechanism; it trains a joint probability distribution of data and labels. The "inner maximization" (the adversarial attack) acts as a biased form of sampling from the model's distribution, while the "outer minimization" (the model update) serves as the learning step.
  • PGD as a Sampler: Projected Gradient Descent (PGD), typically used to craft adversarial examples, is shown to be a competitive, faster, and more stable alternative to Langevin dynamics (a standard EBM sampling method) during training.
  • The InfoNCE Connection: The authors reveal that InfoNCE, the prevalent loss function for unsupervised contrastive learning, is actually an "importance sampling" approximation of their NP-CEM. This provides a long-sought probabilistic foundation for why contrastive learning works.

Novel Sampling Algorithms

Based on this framework, the authors introduce two new sampling algorithms requiring fewer than 50 steps (vastly more efficient than the 1,000+ steps often needed by score-based or diffusion models):

  • Reinforced Conditional Sampling (RCS): For supervised models, this improves image quality by injecting a "positive" gradient that pushes samples toward the data manifold.
  • MaxEnt (Maximum Entropy) Sampling: A novel approach to generate images from unsupervised models by maximizing the entropy of a batch of samples in the feature space.

Key Results

The paper demonstrates that robust models can compete with dedicated generative architectures on the CIFAR-10 dataset.

Comparison of generative performance (IS and FID) across various methods, highlighting the competitive Inception Scores of the proposed CEM models.

As detailed in the table above, the CEM framework achieves impressive generative metrics:

  • Supervised Generation: The supervised CEM on ResNet-50 achieves an Inception Score (IS) of 9.80. This is on par with dedicated generators like StyleGAN2-ADA (10.02) and significantly outperforms previous AT-based generators like TA (7.5).
  • Unsupervised Generation: The unsupervised CEM reaches an IS of 9.61, outperforming previous energy-based models like JEM (8.76).
  • Robustness: Beyond generation, the authors introduced a new regularization term (AT+CR) derived from their framework. It achieves 57.0% adversarial accuracy on WideResNet-34, matching state-of-the-art methods like TRADES while maintaining superior natural accuracy.

Limitations and Future Work

While the Inception Scores (IS) are highly competitive, indicating excellent semantic clarity and global structure, the Fréchet Inception Distance (FID) scores for CEM models (e.g., 40.25 - 55.91) trail behind those of diffusion and score-based models (e.g., NCSN++ at 2.20). This mismatch suggests that while the models understand the global data distribution, there are still local artifacts or textural differences compared to real data.

Future work will likely focus on refining these local textures to improve FID and scaling the CEM framework to higher-resolution datasets like ImageNet, where EBMs traditionally face stability challenges.

Summary updated

Abstract

Adversarial Training (AT) is known as an effective approach to enhance the robustness of deep neural networks. Recently researchers notice that robust models with AT have good generative ability and can synthesize realistic images, while the reason behind it is yet under-explored. In this paper, we demystify this phenomenon by developing a unified probabilistic framework, called Contrastive Energy-based Models (CEM). On the one hand, we provide the first probabilistic characterization of AT through a unified understanding of robustness and generative ability. On the other hand, our unified framework can be extended to the unsupervised scenario, which interprets unsupervised contrastive learning as an important sampling of CEM. Based on these, we propose a principled method to develop adversarial learning and sampling methods. Experiments show that the sampling methods derived from our framework improve the sample quality in both supervised and unsupervised learning. Notably, our unsupervised adversarial sampling method achieves an Inception score of 9.61 on CIFAR-10, which is superior to previous energy-based models and comparable to state-of-the-art generative models.
BibTeX
@inproceedings{wang2022unified,
  author = {Yifei Wang and Yisen Wang and Jiansheng Yang and Zhouchen Lin},
  title = {{A Unified Contrastive Energy-based Model for Understanding the Generative Ability of Adversarial Training}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=XhF2VOMRHS},
}
Figure