HyperGAN: A Generative Model for Diverse, Performant Neural Networks

Neale Ratzlaff, Fuxin Li

TL;DR

HyperGAN is a generative model designed to produce the entire weight set of a neural network in a single forward pass. By learning a distribution of effective parameters rather than a single point estimate, it allows researchers to instantly sample diverse, high-performing ensembles that offer superior uncertainty estimates and adversarial robustness compared to traditional training methods.

What Problem Does This Paper Address?

Standard deep neural networks are notoriously "overconfident"—they often assign high probability to incorrect predictions, especially when faced with data they haven't seen before (out-of-distribution data). While Bayesian Deep Learning (BDL) and deep ensembles can mitigate this by modeling uncertainty, they come with heavy costs:

  1. Computational Expense: Training an ensemble of 100 networks usually requires 100 separate training runs.
  2. Restrictive Assumptions: Many BDL methods (like Variational Inference) rely on simplified "prior" distributions that may not capture the true complexity of the parameter space.
  3. Lack of Diversity: Methods like Dropout only explore a small neighborhood around a single initialization.

HyperGAN aims to solve these issues by using a Generative Adversarial Network (GAN) framework to learn the "manifold" of functional neural network weights, enabling the rapid generation of diverse, accurate models.

Methods

The HyperGAN architecture consists of three main components designed to generate functional weights efficiently:

HyperGAN Architecture
  • The Mixer ($Q$): This is the core innovation. It takes a simple noise vector (from a Gaussian distribution) and projects it into a latent space where dimensions are correlated. This is crucial because the weights of Layer 1 in a neural network must "align" with Layer 2 to produce a valid functional model.
  • Layer-wise Generators ($G$): To keep the model scalable, the authors avoid using one giant generator. Instead, they use parallel generators—one for each layer of the target network—that take the correlated codes from the Mixer as input.
  • The Discriminator ($D$): Unlike a standard GAN that looks at images, this discriminator operates in the latent space. It ensures the Mixer's output stays well-distributed and doesn't "collapse" into generating the same network over and over again (mode collapse).

The Training Process: Instead of training on a dataset of pre-trained weights, HyperGAN is trained directly on the target task (e.g., image classification). It generates a network, tests it on a mini-batch of data, and backpropagates the task loss (like Cross-Entropy) through the generators and the mixer.

Key Insights and Ideas

  • Correlation is Key: The authors found that generating layers independently leads to poor performance. The "Mixer" is essential because it learns the dependencies between different layers of a network.
  • Diversity via Latent Regularization: By forcing the latent codes to match a high-entropy prior using a discriminator, HyperGAN produces networks that are functionally similar (all good at the task) but parametrically diverse.
  • Efficiency: Once trained, sampling a new, fully-trained network is nearly instantaneous. You can create a 1,000-model ensemble in seconds, which would be impossible with standard training.
  • Implicit Bayesianism: The paper argues that by minimizing the Maximum Likelihood Estimate (MLE) error across a distribution of generated weights, HyperGAN is effectively minimizing the KL-divergence between the generated distribution and the true posterior of the weights.

Key Results

HyperGAN was evaluated on standard benchmarks including MNIST, CIFAR-10, and 1D Toy Regression tasks, yielding impressive results:

  • Competitive Accuracy: On MNIST, a 100-network HyperGAN ensemble achieved 99.31% accuracy, outperforming standard training (99.14%) and other Bayesian methods like APD and MC Dropout.
  • Superior Uncertainty: In out-of-distribution (OOD) tests (e.g., testing on notMNIST after training on MNIST), HyperGAN models showed significantly higher predictive entropy than standard ensembles. This means HyperGAN "knows what it doesn't know."
Predictive Entropy on notMNIST
  • Adversarial Robustness: HyperGAN ensembles were much harder to fool with adversarial attacks (like FGSM and PGD). Because the ensemble is so diverse, an attack that fools one generated network is unlikely to fool the others.
  • Scalability: The authors successfully generated weights for convolutional networks on CIFAR-10 using a single GPU with only 4GB of VRAM.

Limitations and Future Work

  • Architecture Constraints: Currently, the HyperGAN itself is composed of MLPs. Scaling this to generate weights for massive architectures like ResNet-101 or Transformers might lead to a "parameter explosion" in the generators themselves.
  • Training Stability: Like all GAN-based approaches, HyperGAN requires careful tuning of the adversarial loss to balance diversity and accuracy.
  • Theoretical Grounding: While the authors provide a link to MLE and KL-divergence, further work is needed to fully understand the theoretical properties of the learned weight distribution compared to a true Bayesian posterior.

Summary updated

Abstract

We introduce HyperGAN, a generative model that learns to generate all the parameters of a deep neural network. HyperGAN first transforms low dimensional noise into a latent space, which can be samp...
BibTeX
@inproceedings{ratzlaff2019hypergan,
  author = {Neale Ratzlaff and Fuxin Li},
  title = {{HyperGAN: A Generative Model for Diverse, Performant Neural Networks}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=SybOFoWO-H},
}
Figure