Neural Rendering Model: Joint Generation and Prediction for Semi-Supervised Learning

Nhat Ho, Tan M. Nguyen, Ankit B. Patel, Anima Anandkumar, Michael I. Jordan, Richard G. Baraniuk

Abstract

Inspired by the success of Convolutional Neural Networks (CNNs) for supervised prediction in images, we design the Deconvolutional Generative Model (DGM), a new probabilistic generative model whose inference calculations correspond to those in a given CNN architecture. The DGM uses a CNN to design the prior distribution in the probabilistic model. Furthermore, the DGM generates images from coarse to finer scales. It introduces a small set of latent variables at each scale, and enforces dependencies among all the latent variables via a conjugate prior distribution. This conjugate prior yields a new regularizer based on paths rendered in the generative model for training CNNs—the Rendering Path Normalization (RPN). We demonstrate that this regularizer improves generalization, both in theory and in practice. In addition, likelihood estimation in the DGM yields training losses for CNNs, and inspired by this, we design a new loss termed as the Max-Min cross entropy which outperforms the traditional cross-entropy loss for object classification. The Max-Min cross entropy suggests a new deep network architecture, namely the Max-Min network, which can learn from less labeled data while maintaining good prediction performance. Our experiments demonstrate that the DGM with the RPN and the Max-Min architecture exceeds or matches the-state-of-art on benchmarks including SVHN, CIFAR10, and CIFAR100 for semi-supervised and supervised learning tasks.

Introduction

While Convolutional Neural Networks (CNNs) are the gold standard for image classification, they are primarily discriminative models. They learn to map inputs to labels but do not inherently model the underlying process of image formation. This limitation often makes them data-hungry and less effective in semi-supervised settings where labeled data is scarce.

This research bridges the gap between discriminative CNNs and generative models by introducing the Deconvolutional Generative Model (DGM). By treating a CNN as the inference step of a specific Bayesian generative process, the authors derive new tools for regularization and loss calculation that significantly improve performance in both supervised and semi-supervised learning.

The Deconvolutional Generative Model (DGM)

The DGM describes a top-down "rendering" process that mirrors the bottom-up "inference" process of a CNN. Instead of going from pixels to labels, the DGM starts with a high-level object template and progressively adds finer details across layers.

The DGM framework: (a) illustrates the symmetry between CNN inference and DGM generation, while (b) presents the underlying probabilistic graphical model.

Key Components of the DGM:

  1. Coarse-to-Fine Rendering: The model generates images hierarchically, moving from abstract class templates to detailed pixel-level representations.
  2. Latent Variables: At each layer, the model utilizes latent variables to govern the rendering. Specifically, $s(\ell)$ determines if a feature should be rendered, while $t(\ell)$ determines where it should be placed.
  3. Probabilistic Interpretation of CNN Layers: A core contribution of this work is the mathematical proof that standard CNN operations are actually forms of Bayesian inference. For instance, the ReLU activation function corresponds to inferring feature existence, and Max-Pooling corresponds to inferring feature location.

Novel Regularization and Loss Functions

By grounding CNNs in a generative framework, the authors derive two novel components for network training:

  • Rendering Path Normalization (RPN): This regularizer is derived from the DGM’s conjugate prior. It penalizes the complexity of "rendering paths," forcing the network to learn more compact and efficient representations. Theoretical analysis shows this provides a generalization bound based on the number of active rendering paths.
  • Max-Min Cross-Entropy: Standard cross-entropy only maximizes the probability of the correct label. The DGM framework suggests a "Max-Min" approach where the network explicitly minimizes the probability of incorrect labels. This is implemented via a dual-branch "Max-Min network" that captures features explicitly absent in certain classes using Min-Pooling and Negative ReLUs.
Mapping between DGM rendering components and standard CNN operations.

Experimental Results

The model was evaluated on standard benchmarks including MNIST, SVHN, CIFAR-10, CIFAR-100, and ImageNet.

Semi-Supervised Performance

On CIFAR-10 with only 4,000 labeled images, the DGM (combined with RPN and Max-Min loss) achieved an error rate of 11.81%, matching or exceeding state-of-the-art methods like Mean Teacher. The model showed particular robustness in low-data regimes, such as SVHN with only 250 to 1,000 labels.

Supervised Performance

The Max-Min cross-entropy loss also improved performance in fully supervised tasks. When applied to a 29-layer ResNet on CIFAR-10, it reduced the test error from a baseline of 2.56% to 2.30%.

Conclusion

The DGM provides a rigorous Bayesian perspective on CNNs, revealing that common architectural choices like ReLU and Max-Pooling are not just heuristics but have clear probabilistic meanings. By unifying generation and prediction, the DGM framework enables more efficient learning from limited data through Rendering Path Normalization and Max-Min architectures. Future work may involve integrating adversarial losses to improve the visual quality of the generated images while maintaining high classification accuracy.

Summary updated

Abstract

Inspired by the success of Convolutional Neural Networks (CNNs) for supervised prediction in images, we design the Deconvolutional Generative Model (DGM), a new probabilistic generative model whose inference calculations correspond to those in a given CNN architecture. The DGM uses a CNN to design the prior distribution in the probabilistic model. Furthermore, the DGM generates images from coarse to finer scales. It introduces a small set of latent variables at each scale, and enforces dependencies among all the latent variables via a conjugate prior distribution. This conjugate prior yields a new regularizer based on paths rendered in the generative model for training CNNs-the Rendering Path Normalization (RPN). We demonstrate that this regularizer improves generalization, both in theory and in practice. In addition, likelihood estimation in the DGM yields training losses for CNNs, and inspired by this, we design a new loss termed as the Max-Min cross entropy which outperforms the traditional cross-entropy loss for object classification. The Max-Min cross entropy suggests a new deep network architecture, namely the Max-Min network, which can learn from less labeled data while maintaining good prediction performance. Our experiments demonstrate that the DGM with the RPN and the Max-Min architecture exceeds or matches the-state-of-art on benchmarks including SVHN, CIFAR10, and CIFAR100 for semi-supervised and supervised learning tasks.
BibTeX
@inproceedings{ho2018neural,
  author = {Nhat Ho and Tan M. Nguyen and Ankit B. Patel and Anima Anandkumar and Michael I. Jordan and Richard G. Baraniuk},
  title = {{Neural Rendering Model: Joint Generation and Prediction for Semi-Supervised Learning}},
  booktitle = {CaltechAUTHORS},
  year = {2018},
}
Figure