Latent Bernoulli Autoencoder

Jiri Fajtl, Vasileios Argyriou, Dorothy Monekosso, Paolo RemagninoICML 2020

While the deep learning world has largely embraced continuous, real-valued latent spaces, there is a compelling case for discreteness. Binary representations are not only more hardware-friendly for compression and memory addressing, but they also mirror the "all-or-nothing" firing patterns of biological neurons. However, training autoencoders with binary latents is notoriously difficult because the "step" function used for binarization has a gradient of zero almost everywhere, effectively killing backpropagation.

In this paper from ICML 2020, researchers introduce the Latent Bernoulli Autoencoder (LBAE), a deterministic model that proves you can have your binary cake and eat it too. The LBAE achieves state-of-the-art performance and, surprisingly, smooth interpolation in a discrete space.

TL;DR

The Latent Bernoulli Autoencoder (LBAE) is a deterministic model that learns multivariate Bernoulli (binary) representations using a straight-through gradient estimator. By applying a novel "random hyperplane rounding" technique inspired by combinatorial optimization, the authors enable smooth interpolation and high-quality image generation within a discrete latent space, outperforming many traditional Variational Autoencoders (VAEs).

What Problem Does This Paper Address?

Training an autoencoder to produce binary latents (e.g., a string of -1s and 1s) presents two major hurdles:

  1. The Gradient Vanishing Problem: Standard gradient descent requires differentiable functions. The binarization operation is a non-differentiable "jump," making it impossible for the decoder's error to tell the encoder how to improve.
  2. The Interpolation Gap: In a continuous space (like a standard VAE), you can "walk" from one point to another to see a smooth transition between images. In a binary space, you usually just have "flips"—change one bit, and the image might jump abruptly.

The authors set out to build a model that trains end-to-end and treats the binary latent space as a structured geometric environment rather than just a collection of independent bits.

Methods

The LBAE architecture consists of a standard encoder-decoder pair with a specialized "bottleneck" in the middle to handle the discrete representations.

LBAE Architecture with Straight-Through Estimator

The Training Trick: Straight-Through Estimator

To get gradients through the binary layer, the authors use a Straight-Through Estimator (STE). During the forward pass, the model binarizes the data into discrete states. During the backward pass, it pretends the binarization was a simple identity function, allowing gradients to flow back to the encoder. Crucially, they add a tanh activation before binarization to limit "optimization overshoot" and stabilize the training process.

Random Hyperplane Rounding

This is the paper's core innovation for sampling and interpolation. Instead of assuming bits are independent, they model the latent distribution using its first two moments (mean and covariance).

Random Hyperplane Rounding for Binary Sampling
  • They map each latent dimension to a unit vector on a high-dimensional hypersphere.
  • The angle between these vectors represents their correlation.
  • To generate a sample, they "slice" the sphere with a random hyperplane. Dimensions on one side of the slice become 1, and those on the other become -1.

Smooth Interpolation

By using Spherical Linear Interpolation (SLERP) to rotate the "slicing" hyperplane between two points on the sphere, they achieve smooth transitions in the decoded image space, even though the underlying latents remain strictly binary.

Datasets and Evaluation

The authors evaluated the LBAE on three standard computer vision benchmarks:

  • MNIST: Hand-written digits (32x32).
  • CIFAR-10: Small natural images (32x32).
  • CelebA: Large-scale face attributes (64x64).

The model was compared against a wide array of competitors, including VAEs, Wasserstein Autoencoders (WAE), and Vector-Quantized VAEs (VQ-VAE).

Key Insights and Results

  • Determinism Wins: Unlike VAEs, which are stochastic and often produce blurry images due to the "noise" injected during training, the LBAE is deterministic. This leads to significantly sharper reconstructions. LBAE achieved the lowest (best) reconstruction FID (Fréchet Inception Distance) and KID scores across all datasets. For example, on CelebA, LBAE's reconstruction FID was 7.71, compared to 39.12 for a standard VAE.
  • Correlations Matter: The "Aha!" moment is that binary bits in a latent space shouldn't be treated as independent. By capturing the pairwise correlations between bits and using the hyperplane rounding method, the model can generate novel samples that look realistic rather than like a jumble of disconnected features.
  • Binary Interpolation is Geometric: The paper demonstrates that "smoothness" is a property of how you navigate the space. By treating the binary latent vector as a result of a geometric partition of a sphere, you can transition between images without the "flickering" or abrupt jumps typical of discrete models.
  • Attribute Manipulation: The authors showed they could successfully add attributes (like eyeglasses or goatees) to faces by simply flipping specific bits identified through the learned mean/covariance of those attributes in the latent space.

Limitations and Future Work

  • Training Speed: The authors note that training is slower than a standard VAE because the gradient propagation through the tanh and binarization layers requires more epochs to converge.
  • Post-hoc Distribution Learning: Currently, the model learns the latent distribution after the autoencoder is trained. A potential future direction is to incorporate the distribution learning into the end-to-end training process itself.
  • Scaling: While results on 64x64 images are impressive, testing the "hyperplane rounding" efficiency on much higher-dimensional latent spaces (for high-res images) remains an open challenge.

Summary updated

BibTeX
@inproceedings{fajtl2020latent,
  author = {Jiri Fajtl and Vasileios Argyriou and Dorothy Monekosso and Paolo Remagnino},
  title = {{Latent Bernoulli Autoencoder}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
}
Figure