A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model

Tan M. Nguyen, Nhat Ho, Ankit Patel, Anima Anandkumar, Michael I. Jordan, Richard G. Baraniuk

Abstract

Inspired by the success of Convolutional Neural Networks (CNNs) for supervised prediction in images, we design the Deconvolutional Generative Model (DGM), a new probabilistic generative model whose inference calculations correspond to those in a given CNN architecture. The DGM uses a CNN to design the prior distribution in the probabilistic model. Furthermore, the DGM generates images from coarse to finer scales. It introduces a small set of latent variables at each scale, and enforces dependencies among all the latent variables via a conjugate prior distribution. This conjugate prior yields a new regularizer based on paths rendered in the generative model for training CNNs-the Rendering Path Normalization (RPN). We demonstrate that this regularizer improves generalization, both in theory and in practice. In addition, likelihood estimation in the DGM yields training losses for CNNs, and inspired by this, we design a new loss termed as the Max-Min cross entropy which outperforms the traditional cross-entropy loss for object classification. The Max-Min cross entropy suggests a new deep network architecture, namely the Max-Min network, which can learn from less labeled data while maintaining good prediction performance. Our experiments demonstrate that the DGM with the RPN and the Max-Min architecture exceeds or matches the-state-of-art on benchmarks including SVHN, CIFAR10, and CIFAR100 for semi-supervised and supervised learning tasks.

Introduction

Convolutional Neural Networks (CNNs) are the standard for supervised image classification, yet they are often criticized as "black boxes" lacking a rigorous probabilistic foundation. This lack of a generative "story" makes it difficult to perform effective semi-supervised learning, where leveraging unlabeled data is essential. Existing generative models like GANs or VAEs often require separate, complex architectures that do not always translate to improved classification performance.

This paper addresses this gap by introducing the Deconvolutional Generative Model (DGM). By "reverse-engineering" CNNs into a generative process, the authors provide a Bayesian grounding for standard deep learning operations, leading to novel regularizers and architectures that significantly boost performance.

The Deconvolutional Generative Model (DGM)

The DGM is a probabilistic framework that generates images from coarse to fine scales. Starting with a high-level object category, the model progressively "renders" an image through multiple layers. At each layer, it introduces latent variables that determine what feature to render (switching variables) and where to place it (translation variables).

The DGM framework illustrates how generation (coarse-to-fine) mirrors standard CNN inference (fine-to-coarse).

Derived from the DGM’s conjugate prior, a new regularizer called Rendering Path Normalization (RPN) penalizes the number of possible "rendering paths." This encourages the network to be more compact and efficient in its image representation, improving generalization in both theory and practice.

Bridging Generation and Inference

A core insight of this work is the formal proof that standard CNN operations are actually Maximum A Posteriori (MAP) inference steps for the DGM. This mapping reveals that common non-linearities are not arbitrary but are mathematically optimal for specific generative tasks:

  • ReLU corresponds to inferring "switching" latent variables (deciding if a feature exists).
  • Max-Pooling corresponds to inferring "translation" latent variables (finding a feature's location).
  • Convolutions act as the mathematical transposes of the DGM’s rendering templates.
This table summarizes the mathematical mapping between DGM generative components and standard CNN operations.

The Max-Min Architecture

Inspired by the DGM's likelihood estimation, the authors propose the Max-Min Network. While a standard CNN (the "Max" branch) uses ReLU and Max-Pooling to identify present features, the "Min" branch uses Negative ReLUs ($min(0, \cdot)$) and Min-Pooling. This dual-branch architecture, paired with a Max-Min cross-entropy loss, forces the network to learn what incorrect classes look like, leading to superior feature separation when labeled data is scarce.

Key Results and Performance

The DGM framework was evaluated across several standard computer vision benchmarks, demonstrating significant gains in both semi-supervised and supervised settings:

  • Semi-Supervised SOTA: On CIFAR-10 with only 1,000 labeled images, the DGM combined with the Max-Min architecture achieved an error rate of 19.79%, outperforming standard CNN baselines.
  • Supervised Gains: Utilizing the Max-Min cross-entropy loss on a 29-layer ResNet (Shake-Shake) achieved a 2.30% error rate on CIFAR-10, a meaningful improvement over the 2.56% baseline.
  • Robustness: The model consistently matched or exceeded state-of-the-art results on SVHN and CIFAR-100, proving that the Bayesian perspective provides a more robust training signal than discriminative training alone.

Conclusion and Future Work

By viewing a CNN as an inference engine for a generative model, the DGM provides a path toward more interpretable and efficient deep learning. While the model currently focuses on classification robustness rather than photorealistic image generation, it opens several avenues for future research. These include integrating adversarial losses to improve the visual quality of rendering templates and investigating better optimization strategies to avoid local minima during the inference process.

Summary updated

Abstract

Inspired by the success of Convolutional Neural Networks (CNNs) for supervised prediction in images, we design the Deconvolutional Generative Model (DGM), a new probabilistic generative model whose inference calculations correspond to those in a given CNN architecture. The DGM uses a CNN to design the prior distribution in the probabilistic model. Furthermore, the DGM generates images from coarse to finer scales. It introduces a small set of latent variables at each scale, and enforces dependencies among all the latent variables via a conjugate prior distribution. This conjugate prior yields a new regularizer based on paths rendered in the generative model for training CNNs-the Rendering Path Normalization (RPN). We demonstrate that this regularizer improves generalization, both in theory and in practice. In addition, likelihood estimation in the DGM yields training losses for CNNs, and inspired by this, we design a new loss termed as the Max-Min cross entropy which outperforms the traditional cross-entropy loss for object classification. The Max-Min cross entropy suggests a new deep network architecture, namely the Max-Min network, which can learn from less labeled data while maintaining good prediction performance. Our experiments demonstrate that the DGM with the RPN and the Max-Min architecture exceeds or matches the-state-of-art on benchmarks including SVHN, CIFAR10, and CIFAR100 for semi-supervised and supervised learning tasks.
BibTeX
@inproceedings{nguyen2018bayesian,
  author = {Tan M. Nguyen and Nhat Ho and Ankit Patel and Anima Anandkumar and Michael I. Jordan and Richard G. Baraniuk},
  title = {{A Bayesian Perspective of Convolutional Neural Networks through a Deconvolutional Generative Model}},
  booktitle = {CaltechAUTHORS},
  year = {2018},
}
Figure