TL;DR
This paper proposes a novel hybrid training framework, HDGE, that unifies supervised learning and contrastive learning by framing them within an energy-based model perspective. By approximating the generative component of this hybrid model with a contrastive loss, HDGE achieves state-of-the-art results in image classification and significantly improves robustness, out-of-distribution detection, and calibration compared to existing methods.
What Problem Does This Paper Address?
The paper addresses the long-standing challenge of effectively combining discriminative and generative modeling approaches in deep learning. While both supervised learning (discriminative) and contrastive learning (often used for representation learning, with generative aspects) have achieved remarkable success independently, they are typically trained separately or simply share a neural network backbone. This paper aims to bridge this gap by proposing a unified framework that leverages the strengths of both paradigms, leading to improved performance across a range of tasks and model properties. Specifically, it tackles the difficulty of training energy-based models (EBMs) due to intractable partition functions and seeks to improve classification accuracy, robustness against adversarial attacks, out-of-distribution detection, and model calibration.
Methods
The core contribution is the Hybrid Discriminative-Generative Energy-based Model (HDGE) framework. HDGE is built upon the insight that supervised learning classifiers can be viewed as energy-based models. The proposed objective function jointly optimizes a discriminative term (standard cross-entropy loss for predicting labels given inputs, $q(y|x)$) and a generative term (predicting inputs given labels, $q(x|y)$).
The key novelty lies in how the generative term is approximated. Instead of relying on computationally expensive sampling methods like SGLD for training EBMs, HDGE approximates the $q(x|y)$ term using a contrastive learning objective. This contrastive loss encourages representations of inputs with the same label to be close while pushing apart representations of inputs with different labels. This approach effectively leverages the benefits of contrastive learning for approximating the generative aspect of the model, making the training more efficient and scalable. The paper also explores a weighted combination of the discriminative and generative losses, controlled by a parameter $\alpha$.
Datasets
The paper uses the following datasets for training and evaluation:
- CIFAR-10 and CIFAR-100: Standard datasets for image classification, used for evaluating classification accuracy, OOD detection, calibration, and adversarial robustness.
- SVHN (Street View House Numbers): Used for OOD detection and robustness evaluation.
- CelebA: Used for OOD detection evaluation.
No new benchmarks were introduced; the focus was on evaluating the proposed method on existing standard datasets.
Key Insights and Ideas
- Unified View of Supervised and Contrastive Learning: The paper elegantly connects supervised learning and contrastive learning by framing them within the context of hybrid discriminative-generative energy-based models. This provides a theoretical foundation for why combining these approaches can be beneficial.
- Contrastive Learning as an EBM Approximation: Using a contrastive loss to approximate the generative component of an energy-based model is a novel and effective strategy. It bypasses the need for expensive sampling methods typically required for training EBMs, leading to more efficient training.
- Synergy of Discriminative and Generative Training: Jointly optimizing both discriminative and generative objectives leads to significant improvements not only in classification accuracy but also in crucial aspects like robustness, out-of-distribution detection, and calibration, suggesting a deeper understanding of the data distribution.
- Scalability Advantage: HDGE offers a computationally feasible alternative to existing EBM training methods, making it more practical for high-dimensional data and wider applications.
Key Results
- Classification Accuracy: HDGE outperforms standard supervised learning, JEM, and Supervised Contrastive Learning on CIFAR-10 and CIFAR-100, achieving state-of-the-art results. For instance, on CIFAR-10, HDGE achieves 96.7% accuracy, compared to 95.8% for supervised learning and 96.4% for the generative-only HDGE.
- Out-of-Distribution Detection: HDGE significantly improves OOD detection performance across various datasets (SVHN, CIFAR-100, CelebA) when using input density $p(x)$ as a score function, consistently outperforming baselines like Glow, IGEBM, and JEM. For example, on SVHN, HDGE achieves an AUROC of 0.96, compared to 0.67 for JEM.
- Calibration: HDGE demonstrates a substantial improvement in calibration, significantly reducing the Expected Calibration Error (ECE) on CIFAR-100 compared to standard supervised learning and JEM, while also maintaining high accuracy.
- Adversarial Robustness: HDGE shows compelling robustness against L2 and L-infinity PGD attacks, achieving comparable or better performance than state-of-the-art adversarial training methods.
- Approximation Impact: Increasing the number of negative samples ($K$) in the contrastive approximation generally improves HDGE's performance, indicating the effectiveness of the approximation strategy.
Limitations and Future Work
- Approximation Guarantees: The paper acknowledges the lack of formal guarantees for the contrastive approximation of the generative loss. The justification relies on experimental results, and further theoretical analysis could be beneficial.
- Computational Cost of Large K: While increasing $K$ improves performance, it also increases memory requirements. The paper uses memory banks to mitigate this, but extremely large $K$ might still be challenging.
- Generative Modeling Enhancement: While HDGE improves classification and other properties, the paper suggests that further enhancing generative modeling performance can be achieved by jointly optimizing the marginal log-likelihood $p(x)$ (as in JEM) alongside the HDGE objective. This hybrid approach (HDGE+JEM) shows promising results in generative tasks.
- Exploration of $\alpha$ Parameter: The paper evaluates different values of $\alpha$ (weighting discriminative and generative terms) and finds $\alpha=0.5$ to be optimal for many tasks, but further systematic study of this parameter could be explored.
Summary updated