Abstract
Deep Hybrid Models (DHMs) bridge the gap between discriminative and generative modeling by coupling them through shared latent variables. Unlike traditional weight-sharing approaches, DHMs utilize a high-level latent representation that acts as a powerful regularizer. This framework achieves state-of-the-art results in semi-supervised learning tasks, particularly on the SVHN dataset, while improving the generalization of standard deep learning classifiers.
Introduction
In modern machine learning, models typically fall into two categories: discriminative and generative. Discriminative models, such as standard Convolutional Neural Networks (CNNs), excel at mapping inputs $x$ to labels $y$ but often require vast amounts of labeled data and are susceptible to overfitting. Conversely, generative models like Variational Auto-Encoders (VAEs) or Generative Adversarial Networks (GANs) learn the underlying data distribution $p(x, y)$, making them robust for semi-supervised learning but often less accurate in pure classification tasks.
Hybridizing these paradigms has historically been challenging. Previous efforts relied on rigid parameter sharing, which often limits the flexibility of complex deep architectures. This paper introduces Deep Hybrid Models (DHMs), which achieve hybridization through latent variable coupling, offering the benefits of both approaches without the constraints of weight sharing.
Methods: Latent Variable Coupling
The core innovation of the DHM framework is the use of a shared latent variable $z$ to couple the discriminative and generative components. Instead of forcing identical weights across networks, the model is defined by the joint distribution:
$$p(x, y, z) = p(y|x, z) \cdot p(x, z)$$
In this architecture, $z$ serves as a high-level feature representation that must simultaneously support two objectives:
- Discriminative: Predicting the label $y$ from input $x$ and features $z$.
- Generative: Reconstructing or generating the input $x$ from the features $z$.
Training is performed using a multi-conditional likelihood objective. This objective incorporates scalar weights, $\alpha$ and $\beta$, allowing for precise control over the balance between discriminative and generative signals. The framework is versatile, supporting both explicit density models (VAEs) and implicit density models (GANs).
Experimental Results
The authors evaluated DHMs on MNIST, SVHN, and CIFAR-10 benchmarks. In semi-supervised settings with limited labels (e.g., 1,000 labels for SVHN), the model demonstrated significant performance gains.
State-of-the-Art Semi-Supervised Learning
On the SVHN dataset, the Implicit DHM achieved a benchmark-setting error rate of 4.45%, outperforming established methods like the $\Pi$-model (5.45%) and ALI (7.42%). The generative task serves as a natural regularizer; by requiring the network to learn features $z$ capable of image reconstruction, the model is less likely to overfit on small labeled subsets.
Improved Generalization and Generative Performance
The coupling mechanism provides mutual benefits. Adding a discriminative signal helps the generative model learn superior features, resulting in higher log-likelihoods than purely unsupervised counterparts. On MNIST, the hybrid model achieved a variational lower bound (ELBO) of -75.5, surpassing previous generative results.
As shown in the performance plots, the hybrid approach (HDGM) consistently maintains or improves upon the discriminative baseline across various strengths of the discriminative objective, particularly on the more complex SVHN dataset.
Discussion and Conclusion
While DHMs show strong performance on digit datasets, challenges remain for more complex data like CIFAR-10, where the generative component (GAN) may struggle to capture high-frequency details. However, the flexibility of latent coupling means that as generative architectures like Diffusion models continue to evolve, they can be seamlessly integrated into the DHM framework. By bridging the gap between discriminative and generative paradigms, DHMs provide a robust path forward for efficient semi-supervised learning and improved model generalization.
Summary updated