Hybrid Deep Discriminative/Generative Models for Semi-Supervised Learning

Volodymyr Kuleshov, Stefano ErmonUAI 2017 · OpenReview

Abstract

Deep Hybrid Models (DHMs) bridge the gap between discriminative and generative modeling by coupling them through shared latent variables. Unlike traditional weight-sharing approaches, DHMs utilize a high-level latent representation that acts as a powerful regularizer. This framework achieves state-of-the-art results in semi-supervised learning tasks, particularly on the SVHN dataset, while improving the generalization of standard deep learning classifiers.

Introduction

In modern machine learning, models typically fall into two categories: discriminative and generative. Discriminative models, such as standard Convolutional Neural Networks (CNNs), excel at mapping inputs $x$ to labels $y$ but often require vast amounts of labeled data and are susceptible to overfitting. Conversely, generative models like Variational Auto-Encoders (VAEs) or Generative Adversarial Networks (GANs) learn the underlying data distribution $p(x, y)$, making them robust for semi-supervised learning but often less accurate in pure classification tasks.

Hybridizing these paradigms has historically been challenging. Previous efforts relied on rigid parameter sharing, which often limits the flexibility of complex deep architectures. This paper introduces Deep Hybrid Models (DHMs), which achieve hybridization through latent variable coupling, offering the benefits of both approaches without the constraints of weight sharing.

Methods: Latent Variable Coupling

The core innovation of the DHM framework is the use of a shared latent variable $z$ to couple the discriminative and generative components. Instead of forcing identical weights across networks, the model is defined by the joint distribution:

$$p(x, y, z) = p(y|x, z) \cdot p(x, z)$$

In this architecture, $z$ serves as a high-level feature representation that must simultaneously support two objectives:

  1. Discriminative: Predicting the label $y$ from input $x$ and features $z$.
  2. Generative: Reconstructing or generating the input $x$ from the features $z$.
Architecture of the Deep Hybrid Model showing the coupling of CNN and VAE components through a shared latent space z.

Training is performed using a multi-conditional likelihood objective. This objective incorporates scalar weights, $\alpha$ and $\beta$, allowing for precise control over the balance between discriminative and generative signals. The framework is versatile, supporting both explicit density models (VAEs) and implicit density models (GANs).

Experimental Results

The authors evaluated DHMs on MNIST, SVHN, and CIFAR-10 benchmarks. In semi-supervised settings with limited labels (e.g., 1,000 labels for SVHN), the model demonstrated significant performance gains.

State-of-the-Art Semi-Supervised Learning

On the SVHN dataset, the Implicit DHM achieved a benchmark-setting error rate of 4.45%, outperforming established methods like the $\Pi$-model (5.45%) and ALI (7.42%). The generative task serves as a natural regularizer; by requiring the network to learn features $z$ capable of image reconstruction, the model is less likely to overfit on small labeled subsets.

Improved Generalization and Generative Performance

The coupling mechanism provides mutual benefits. Adding a discriminative signal helps the generative model learn superior features, resulting in higher log-likelihoods than purely unsupervised counterparts. On MNIST, the hybrid model achieved a variational lower bound (ELBO) of -75.5, surpassing previous generative results.

Discriminative performance of HDGM on MNIST and SVHN across different discriminative strengths.

As shown in the performance plots, the hybrid approach (HDGM) consistently maintains or improves upon the discriminative baseline across various strengths of the discriminative objective, particularly on the more complex SVHN dataset.

Discussion and Conclusion

While DHMs show strong performance on digit datasets, challenges remain for more complex data like CIFAR-10, where the generative component (GAN) may struggle to capture high-frequency details. However, the flexibility of latent coupling means that as generative architectures like Diffusion models continue to evolve, they can be seamlessly integrated into the DHM framework. By bridging the gap between discriminative and generative paradigms, DHMs provide a robust path forward for efficient semi-supervised learning and improved model generalization.

Summary updated

BibTeX
@inproceedings{kuleshov2017hybrid,
  author = {Volodymyr Kuleshov and Stefano Ermon},
  title = {{Hybrid Deep Discriminative/Generative Models for Semi-Supervised Learning}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2017},
  url = {https://openreview.net/forum?id=Easke2exB4X},
}
Figure