β-VAEs can retain label information even at high compression

Emily Fertig, Aryan Arbabi, Alexander A. AlemiarXiv 2018 · arXiv

Abstract

β-Variational Autoencoders (β-VAEs) are typically trained using an unsupervised reconstruction objective. This study investigates whether such models naturally prioritize semantic information, such as class labels, when subjected to high compression. By introducing a metric called Label Distortion, we demonstrate that β-VAEs can retain nearly all label information even at high compression rates, significantly outperforming linear baselines like PCA. Our findings suggest that the unsupervised objective of VAEs is surprisingly well-aligned with human-relevant features, particularly when using specific architectural choices like factorized decoders.

Introduction

In unsupervised learning, the goal is to discover the underlying structure of data without explicit labels. A Variational Autoencoder (VAE) achieves this by compressing data into a latent bottleneck and reconstructing it. This paper explores a fundamental question: when forced to be extremely selective due to high compression, does a VAE prioritize "important" semantic features?

Using a "red ball" analogy, if a model selects a few bits of information from a high-dimensional image but manages to preserve the bits necessary for classification, it indicates a non-random preference for salient features. This study quantifies this "salience bias" in β-VAEs trained on MNIST and Omniglot.

Methods

The researchers employ the β-VAE framework, where the hyperparameter β balances reconstruction fidelity (Distortion, $D$) and compression (Rate, $R$). To measure semantic retention, they introduce Label Distortion ($C$). This is calculated by freezing a pre-trained unsupervised encoder and training a separate variational classifier to recover labels from the latent representations.

The performance is benchmarked against several baselines, including Stochastic PCA and randomly initialized networks, to isolate the impact of the learning objective from architectural inductive biases.

Results

The experimental results across MNIST and Omniglot datasets reveal several key insights into the behavior of β-VAEs.

Label distortion vs. Rate and Image distortion for MNIST and Omniglot datasets across various model configurations.

The Salience Bias

On the MNIST dataset, β-VAEs consistently outperform baselines, retaining nearly all label information even at compression factors exceeding 9x. This suggests a clear preference for filtering out noise while preserving human-salient features like digit identity. As shown in the figures, β-VAEs (various markers) maintain lower label distortion at low rates compared to the PCA baseline.

Impact of Decoder Strength

A counter-intuitive finding is that "weaker" (factorized) decoders often result in better label retention in the encoder. Powerful autoregressive decoders can "fill in" details, allowing the encoder to neglect global semantic features. Conversely, a weaker decoder forces the encoder to capture essential shared features across pixels to minimize reconstruction error.

Dataset Complexity

While the salience bias is strong in MNIST, it is less pronounced in the more complex Omniglot dataset. This indicates that as data complexity increases, standard Gaussian priors may struggle, and more flexible models might be required to achieve similar levels of semantic filtering. For the richer Omniglot dataset, the model's ability to retain information about both characters and alphabets was tested, showing higher baseline label distortion than MNIST.

Conclusion

This work demonstrates that β-VAEs possess an inherent "salience bias," allowing them to discover and prioritize semantic labels without supervision. While the effect is most prominent in simpler datasets like MNIST, the ability of these models to outperform linear methods like PCA highlights the efficiency of non-linear latent features. Future work will explore more complex datasets like CelebA and alternative generative architectures to further understand the limits of unsupervised semantic discovery.

Summary updated

Abstract

In this paper, we investigate the degree to which the encoding of a $\beta$-VAE captures label information across multiple architectures on Binary Static MNIST and Omniglot. Even though they are trained in a completely unsupervised manner, we demonstrate that a $\beta$-VAE can retain a large amount of label information, even when asked to learn a highly compressed representation.
BibTeX
@misc{fertig2018vaes,
  author = {Emily Fertig and Aryan Arbabi and Alexander A. Alemi},
  title = {{β-VAEs can retain label information even at high compression}},
  year = {2018},
  eprint = {1812.02682},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.02682},
}
Figure