Batch norm with entropic regularization turns deterministic autoencoders into generative models

Amur Ghose, Abdullah Rashwan, Pascal Poupart

Abstract

The variational autoencoder is a well defined deep generative model that utilizes an encoder-decoder framework where an encoding neural network outputs a non-deterministic code for reconstructing an input. The encoder achieves this by sampling from a distribution for every input, instead of outputting a deterministic code per input. The great advantage of this process is that it allows the use of the network as a generative model for sampling from the data distribution beyond provided samples for training. We show in this work that utilizing batch normalization as a source for non-determinism suffices to turn deterministic autoencoders into generative models on par with variational ones, so long as we add a suitable entropic regularization to the training objective.

Introduction

Generative modeling has traditionally relied on stochasticity to bridge the gap between compression and synthesis. Variational Autoencoders (VAEs) introduce this through a sampling layer in the latent space, typically using the reparameterization trick to remain differentiable. While effective, VAEs often suffer from "posterior collapse" and produce blurry reconstructions due to the constraints of the KL-divergence term. Deterministic Autoencoders (AEs) produce sharper results but lack a structured latent space suitable for random sampling.

This paper introduces the Entropic Autoencoder (EAE), a novel approach that transforms a standard deterministic AE into a generative model. By leveraging Batch Normalization (BN) and entropic regularization, the model forces the latent space to follow a Gaussian distribution without requiring explicit sampling during training.

Methodology: The Maximum Entropy Principle

The core innovation of the EAE lies in combining Batch Normalization with an entropic regularizer. According to the Maximum Entropy Principle, a distribution with a fixed mean and variance that maximizes its entropy is a Gaussian.

The EAE architecture implements this as follows:

  1. Batch Normalization (BN): A BN layer is placed at the end of the encoder. By disabling affine parameters (shift and scale), the layer strictly fixes the latent codes' mean to 0 and variance to 1.
  2. Entropic Regularization: An additional term is added to the loss function to maximize the entropy ($H$) of the latent codes within each minibatch.
  3. Entropy Estimation: To maintain a deterministic training loop, the authors use the Kozachenko-Leonenko estimator. This non-parametric method estimates entropy based on the Euclidean distances between nearest neighbors in the latent space.

By fixing the first two moments via BN and maximizing entropy, the latent space naturally organizes into a spherical Gaussian, making it "sample-ready" for generation.

Experimental Results

The researchers evaluated the EAE on MNIST, CIFAR-10, and CelebA datasets, comparing it against standard VAEs and Regularized Autoencoders (RAEs).

Qualitative Analysis

On the MNIST dataset, the EAE demonstrates the ability to generate clear, diverse handwritten digits by sampling from the latent space. This confirms that the deterministic bottleneck has successfully learned a continuous, generative manifold.

Generated MNIST digits showing the model's generative capability.

Quantitative Performance

The EAE consistently outperformed VAEs in terms of Fréchet Inception Distance (FID), which measures both image quality and diversity. On CIFAR-10, EAE achieved an FID of 85.26, significantly better than the standard VAE's 106.37. On CelebA, EAE achieved an FID of 44.63, outperforming VAE's 48.12.

Generated CIFAR-10 samples demonstrating image sharpness.

Visual results on CIFAR-10 show that EAEs produce sharper images with fewer artifacts compared to traditional variational approaches, while maintaining the ease of sampling found in generative models.

Discussion and Conclusion

The EAE provides a "backdoor" to generative modeling by using the mathematical properties of entropy rather than explicit stochastic sampling. This approach avoids the complexities of posterior collapse and the blurriness often associated with VAEs. While the current implementation benefits from post-hoc density estimation (like fitting a GMM), the inherent Gaussianization of the latent space marks a significant step forward for deterministic generative models.

Summary updated

Abstract

The variational autoencoder is a well defined deep generative model that utilizes an encoder-decoder framework where an encoding neural network outputs a non-deterministic code for reconstructing an input. The encoder achieves this by sampling from a distribution for every input, instead of outputting a deterministic code per input. The great advantage of this process is that it allows the use of the network as a generative model for sampling from the data distribution beyond provided samples for training. We show in this work that utilizing batch normalization as a source for non-determinism suffices to turn deterministic autoencoders into generative models on par with variational ones, so long as we add a suitable entropic regularization to the training objective.
BibTeX
@inproceedings{ghose2020batch,
  author = {Amur Ghose and Abdullah Rashwan and Pascal Poupart},
  title = {{Batch norm with entropic regularization turns deterministic autoencoders into generative models}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=RGlPy58R9g},
}
Figure