Abstract
The variational autoencoder is a well defined deep generative model that utilizes an encoder-decoder framework where an encoding neural network outputs a non-deterministic code for reconstructing an input. The encoder achieves this by sampling from a distribution for every input, instead of outputting a deterministic code per input. The great advantage of this process is that it allows the use of the network as a generative model for sampling from the data distribution beyond provided samples for training. We show in this work that utilizing batch normalization as a source for non-determinism suffices to turn deterministic autoencoders into generative models on par with variational ones, so long as we add a suitable entropic regularization to the training objective.
Introduction
Generative modeling has traditionally relied on stochasticity to bridge the gap between compression and synthesis. Variational Autoencoders (VAEs) introduce this through a sampling layer in the latent space, typically using the reparameterization trick to remain differentiable. While effective, VAEs often suffer from "posterior collapse" and produce blurry reconstructions due to the constraints of the KL-divergence term. Deterministic Autoencoders (AEs) produce sharper results but lack a structured latent space suitable for random sampling.
This paper introduces the Entropic Autoencoder (EAE), a novel approach that transforms a standard deterministic AE into a generative model. By leveraging Batch Normalization (BN) and entropic regularization, the model forces the latent space to follow a Gaussian distribution without requiring explicit sampling during training.
Methodology: The Maximum Entropy Principle
The core innovation of the EAE lies in combining Batch Normalization with an entropic regularizer. According to the Maximum Entropy Principle, a distribution with a fixed mean and variance that maximizes its entropy is a Gaussian.
The EAE architecture implements this as follows:
- Batch Normalization (BN): A BN layer is placed at the end of the encoder. By disabling affine parameters (shift and scale), the layer strictly fixes the latent codes' mean to 0 and variance to 1.
- Entropic Regularization: An additional term is added to the loss function to maximize the entropy ($H$) of the latent codes within each minibatch.
- Entropy Estimation: To maintain a deterministic training loop, the authors use the Kozachenko-Leonenko estimator. This non-parametric method estimates entropy based on the Euclidean distances between nearest neighbors in the latent space.
By fixing the first two moments via BN and maximizing entropy, the latent space naturally organizes into a spherical Gaussian, making it "sample-ready" for generation.
Experimental Results
The researchers evaluated the EAE on MNIST, CIFAR-10, and CelebA datasets, comparing it against standard VAEs and Regularized Autoencoders (RAEs).
Qualitative Analysis
On the MNIST dataset, the EAE demonstrates the ability to generate clear, diverse handwritten digits by sampling from the latent space. This confirms that the deterministic bottleneck has successfully learned a continuous, generative manifold.
Quantitative Performance
The EAE consistently outperformed VAEs in terms of Fréchet Inception Distance (FID), which measures both image quality and diversity. On CIFAR-10, EAE achieved an FID of 85.26, significantly better than the standard VAE's 106.37. On CelebA, EAE achieved an FID of 44.63, outperforming VAE's 48.12.
Visual results on CIFAR-10 show that EAEs produce sharper images with fewer artifacts compared to traditional variational approaches, while maintaining the ease of sampling found in generative models.
Discussion and Conclusion
The EAE provides a "backdoor" to generative modeling by using the mathematical properties of entropy rather than explicit stochastic sampling. This approach avoids the complexities of posterior collapse and the blurriness often associated with VAEs. While the current implementation benefits from post-hoc density estimation (like fitting a GMM), the inherent Gaussianization of the latent space marks a significant step forward for deterministic generative models.
Summary updated