Generative modeling of convolutional neural networks

Jifeng Dai, Yang Lu, Yingnian Wu

Abstract

The convolutional neural networks (ConvNets) have proven to be a powerful tool for discriminative learning. Recently researchers have also started to show interest in the generative aspects of ConvNets in order to gain a deeper understanding of what ConvNets have learned and how to further improve them. This paper investigates generative modeling of ConvNets. The main contributions include: (1) We construct a generative model for the ConvNet in the form of exponential tilting of a reference distribution. (2) We propose a generative gradient for pre-training ConvNets by a non-parametric importance sampling scheme. It is fundamentally different from the commonly used discriminative gradient, and yet shares the same computational architecture and cost as the latter. (3) We propose a generative visualization method for the ConvNets by sampling from an explicit parametric image distribution. The proposed visualization method can directly draw synthetic samples for any given node in a trained ConvNet by the Hamiltonian Monte Carlo algorithm, without resorting to any extra hold-out images. Experiments on the challenging ImageNet benchmark show that the proposed generative gradient pre-training helps improve the performances of ConvNets in both supervised and semi-supervised settings, and the proposed generative visualization method generates meaningful and varied samples of synthetic images from a large and deep ConvNet.

Introduction

Convolutional Neural Networks (ConvNets) are the gold standard for discriminative tasks like image classification. However, they are often treated as "black boxes," where understanding what features they prioritize remains a challenge. Standard discriminative training ($p(y|x)$) focuses primarily on the boundaries between classes. This paper argues that generative modeling ($p(x|y)$)—learning the actual distribution of images within a category—provides a stronger learning signal. By re-imagining ConvNets as generative models, the authors aim to improve feature representation, enhance semi-supervised performance, and provide a more intuitive way to visualize the internal logic of the network.

Methodology

The authors bridge the gap between discriminative and generative models using three key innovations. First, they introduce Exponential Tilting, defining a probability distribution for images where the ConvNet’s scoring function "tilts" a basic reference distribution (like white noise), transforming the ConvNet into an energy-based generative model.

Second, they propose the Generative Gradient. Unlike the standard discriminative gradient, which compares a label to a prediction, the generative gradient compares the observed image to a set of "importance samples." Crucially, they use the training batch itself as a non-parametric reference, making the computational cost identical to standard backpropagation.

The dual nature of the proposed ConvNet model, supporting both bottom-up discriminative and top-down generative processes.

Finally, to "see" what a specific neuron or class represents, they use Hamiltonian Monte Carlo (HMC). This method starts with random noise and uses the network's own gradients to "evolve" that noise into a synthetic image that maximizes the response of a specific node. This top-down approach does not require searching through a database of existing images.

Experiments and Results

The researchers evaluated their approach on MNIST and ImageNet (ILSVRC 2012) using architectures such as AlexNet. On MNIST, combining generative and discriminative gradients (GG+DG) reduced the error rate of LeNet from 1.03% to 0.78%. On ImageNet, generative pre-training improved AlexNet's top-1 accuracy by approximately 1.1%.

The model's performance in semi-supervised learning was particularly notable. On ImageNet with only 50,000 labeled samples, the approach achieved an error rate of 89.2%, significantly outperforming the standard discriminative baseline of 92.2%. Qualitatively, the HMC-generated images reveal that the network learns global structures rather than just local textures.

Synthetic image generation using Hamiltonian Monte Carlo (HMC) at iteration 100, revealing the learned features of a specific class.

Conclusion

While the importance sampling used for the generative gradient becomes less reliable as the model distribution sharpens, it serves as a highly effective pre-training strategy. This generative framework provides a powerful tool for both improving ConvNet performance and interpreting the complex features learned by deep architectures. Future work may involve applying this framework to non-image data and refining sampling algorithms for even deeper models.

Summary updated

Abstract

The convolutional neural networks (ConvNets) have proven to be a powerful tool for discriminative learning. Recently researchers have also started to show interest in the generative aspects of ConvNets in order to gain a deeper understanding of what ConvNets have learned and how to further improve them. This paper investigates generative modeling of ConvNets. The main contributions include: (1) We construct a generative model for the ConvNet in the form of exponential tilting of a reference distribution. (2) We propose a generative gradient for pre-training ConvNets by a non-parametric importance sampling scheme. It is fundamentally different from the commonly used discriminative gradient, and yet shares the same computational architecture and cost as the latter. (3) We propose a generative visualization method for the ConvNets by sampling from an explicit parametric image distribution. The proposed visualization method can directly draw synthetic samples for any given node in a trained ConvNet by the Hamiltonian Monte Carlo algorithm, without resorting to any extra hold-out images. Experiments on the challenging ImageNet benchmark show that the proposed generative gradient pre-training helps improve the performances of ConvNets in both supervised and semi-supervised settings, and the proposed generative visualization method generates meaningful and varied samples of synthetic images from a large and deep ConvNet.
BibTeX
@inproceedings{dai2015generative,
  author = {Jifeng Dai and Yang Lu and Yingnian Wu},
  title = {{Generative modeling of convolutional neural networks}},
  booktitle = {International Conference on Learning Representations},
  year = {2015},
  url = {http://arxiv.org/abs/1412.6296},
}

Loading related content...

Figure