Exact Rate-Distortion in Autoencoders via Echo Noise

Rob Brekelmans, Daniel Moyer, Aram Galstyan, Greg Ver Steeg

Introduction

In representation learning, the goal is to compress an input $X$ into a latent code $Z$ while retaining essential information. This objective is fundamentally a rate-distortion problem: minimizing distortion (reconstruction error) while maintaining a low information rate (compression).

Standard Variational Autoencoders (VAEs) typically rely on Gaussian noise and Gaussian priors to ensure mathematical tractability. However, the KL-divergence term in VAEs serves only as an upper bound on the actual information rate. A significant gap between this bound and the true rate can lead to suboptimal representations. Furthermore, the rigid assumption of a Gaussian latent space may be too restrictive for complex, real-world data distributions.

The Echo Noise Channel

To address these limitations, the authors introduce the Echo Noise channel, defined by a stochastic encoder: $$z = f(x) + S(x)\epsilon$$ where $f(x)$ is the mean and $S(x)$ is a scaling matrix. Unlike standard approaches that draw $\epsilon$ from a fixed distribution, Echo Noise constructs $\epsilon$ as an infinite sum of attenuated "echoes" from other samples within the same dataset.

Comparison of Echo Noise and Gaussian Noise channels.

The core theoretical contribution (Lemma 2.1 & Theorem 2.2) proves that this construction ensures the distribution of noise $\epsilon$ exactly matches the marginal distribution of the latent codes $q(z)$. This alignment causes complex entropy terms in the mutual information formula to cancel out, yielding an exact expression for the information rate: $$I(X;Z) = -E_x \log | \det S(x) |$$ This allows the model to optimize the true information rate directly, bypassing the need for predefined priors or expensive density estimation techniques like Normalizing Flows.

Experimental Setup

The method was evaluated across several benchmarks:

  • Static Binary MNIST & Omniglot: Used for generative modeling and assessing log-likelihood bounds.
  • Fashion MNIST (fMNIST): Used to generate rate-distortion curves.
  • dSprites: A dataset with known generative factors (shape, scale, rotation, position), used to evaluate representation disentanglement.

Key Results

Echo Noise demonstrates several advantages over traditional VAEs and advanced variants like InfoVAE, VampPrior, and Inverse Autoregressive Flows (IAF).

Rate-Distortion curves for Binary MNIST and Omniglot.

Superior Rate-Distortion Trade-offs

As shown in the rate-distortion plots, Echo models consistently "dominate" $\beta$-VAEs, achieving lower distortion for the same level of compression across nearly the entire operational range. This indicates a more efficient use of the latent capacity.

Enhanced Disentanglement

On the dSprites dataset, Echo Noise achieved higher disentanglement scores (using FactorVAE and MIG metrics). Notably, it outperformed standard VAEs when dependencies were introduced between ground-truth factors, as Echo Noise can capture correlations that independent Gaussian priors cannot.

Parameter Efficiency

By using an implicit optimal prior, Echo Noise achieves performance comparable to complex flow-based models but with significantly fewer parameters. It avoids the overhead of training additional transformation networks required by models like IAF.

Discussion and Limitations

While Echo Noise provides a tighter bound on information rate, it faces challenges in the low-rate limit. Numerical stability requirements and the truncation of the infinite sum for sampling make it difficult to reach near-zero rate states. Additionally, constructing the noise involves sampling multiple data points from a batch, introducing a slight computational overhead during training. Future work may explore non-diagonal scaling matrices $S(x)$ to further capture complex dependencies within the latent space.

Summary updated

BibTeX
@inproceedings{brekelmans2019exact,
  author = {Rob Brekelmans and Daniel Moyer and Aram Galstyan and Greg Ver Steeg},
  title = {{Exact Rate-Distortion in Autoencoders via Echo Noise}},
  booktitle = {NeurIPS},
  year = {2019},
  url = {https://openreview.net/forum?id=Icpe66ET2VwU},
}
Figure