Abstract: The variational autoencoder is a well defined deep generative model that utilizes an encoder-decoder framework where an encoding neural network outputs a non-deterministic code for reconstructing an input. The encoder achieves this by sampling from a distribution for every input, instead of outputting a deterministic code per input. The great advantage of this process is that it allows the use of the network as a generative model for sampling from the data distribution beyond provided samples for training. We show in this work that utilizing batch normalization as a source for non-determinism suffices to turn deterministic autoencoders into generative models on par with variational ones, so long as we add a suitable entropic regularization to the training objective.
Introduction
Deep generative models typically fall into two categories: Generative Adversarial Networks (GANs) or Variational Autoencoders (VAEs). VAEs are favored for their structured latent space, but they require stochastic sampling during training, which necessitates the "reparameterization trick" and can lead to posterior collapse—where the model ignores the latent code entirely. Deterministic autoencoders (AEs) are easier to optimize but lack a naturally generative latent space. This paper introduces the Entropic Autoencoder (EAE), which achieves generative capabilities in a deterministic framework by forcing the latent space into a spherical Gaussian distribution.
Methodology
The EAE architecture is a standard deterministic autoencoder enhanced with two specific components:
- Batch Normalization (BN): A BN layer is placed at the end of the encoder to fix the mean of the latent codes to 0 and the variance to 1.
- Entropic Regularization: A term is added to the loss function to maximize the entropy of the latent codes. This is calculated using the non-parametric Kozachenko-Leonenko estimator, based on nearest-neighbor distances in a minibatch.
This approach leverages the Maximum Entropy Principle: a distribution with a fixed mean and variance that has the maximum possible entropy is a Gaussian. By fixing the moments with BN and maximizing entropy, the model naturally organizes its latent space into a spherical Gaussian without requiring sampling during training.
Experimental Results
The EAE was evaluated on MNIST, CIFAR-10, and CelebA benchmarks. It consistently outperformed standard VAEs and previous state-of-the-art deterministic models in terms of Fréchet Inception Distance (FID).
On the CelebA dataset, the EAE achieved an FID of 44.63, outperforming the VAE's 48.12. Visually, the EAE produces sharper images than VAEs, which are often limited by blurry outputs. For CIFAR-10, the EAE reached an FID of 85.26, a significant improvement over the standard VAE (106.37).
One of the most significant advantages of the EAE is that it allows for simple isotropic sampling—generating new data by pulling a random vector from $N(0, I)$ and feeding it to the decoder—without the need for post-hoc density estimation like Gaussian Mixture Models.
Discussion and Conclusion
The EAE demonstrates that the complex stochastic machinery of VAEs is not strictly necessary for high-quality generative modeling. By using batch normalization to fix moments and entropic regularization to maximize distribution spread, deterministic models can achieve competitive generative performance. Future work will focus on optimizing the quadratic complexity of the entropy estimator and exploring integrations with vector quantization to further improve image quality.
Summary updated