Abstract
β-Variational Autoencoders (β-VAEs) are typically trained using an unsupervised reconstruction objective. This study investigates whether such models naturally prioritize semantic information, such as class labels, when subjected to high compression. By introducing a metric called Label Distortion, we demonstrate that β-VAEs can retain nearly all label information even at high compression rates, significantly outperforming linear baselines like PCA. Our findings suggest that the unsupervised objective of VAEs is surprisingly well-aligned with human-relevant features, particularly when using specific architectural choices like factorized decoders.
Introduction
In unsupervised learning, the goal is to discover the underlying structure of data without explicit labels. A Variational Autoencoder (VAE) achieves this by compressing data into a latent bottleneck and reconstructing it. This paper explores a fundamental question: when forced to be extremely selective due to high compression, does a VAE prioritize "important" semantic features?
Using a "red ball" analogy, if a model selects a few bits of information from a high-dimensional image but manages to preserve the bits necessary for classification, it indicates a non-random preference for salient features. This study quantifies this "salience bias" in β-VAEs trained on MNIST and Omniglot.
Methods
The researchers employ the β-VAE framework, where the hyperparameter β balances reconstruction fidelity (Distortion, $D$) and compression (Rate, $R$). To measure semantic retention, they introduce Label Distortion ($C$). This is calculated by freezing a pre-trained unsupervised encoder and training a separate variational classifier to recover labels from the latent representations.
The performance is benchmarked against several baselines, including Stochastic PCA and randomly initialized networks, to isolate the impact of the learning objective from architectural inductive biases.
Results
The experimental results across MNIST and Omniglot datasets reveal several key insights into the behavior of β-VAEs.
The Salience Bias
On the MNIST dataset, β-VAEs consistently outperform baselines, retaining nearly all label information even at compression factors exceeding 9x. This suggests a clear preference for filtering out noise while preserving human-salient features like digit identity. As shown in the figures, β-VAEs (various markers) maintain lower label distortion at low rates compared to the PCA baseline.
Impact of Decoder Strength
A counter-intuitive finding is that "weaker" (factorized) decoders often result in better label retention in the encoder. Powerful autoregressive decoders can "fill in" details, allowing the encoder to neglect global semantic features. Conversely, a weaker decoder forces the encoder to capture essential shared features across pixels to minimize reconstruction error.
Dataset Complexity
While the salience bias is strong in MNIST, it is less pronounced in the more complex Omniglot dataset. This indicates that as data complexity increases, standard Gaussian priors may struggle, and more flexible models might be required to achieve similar levels of semantic filtering. For the richer Omniglot dataset, the model's ability to retain information about both characters and alphabets was tested, showing higher baseline label distortion than MNIST.
Conclusion
This work demonstrates that β-VAEs possess an inherent "salience bias," allowing them to discover and prioritize semantic labels without supervision. While the effect is most prominent in simpler datasets like MNIST, the ability of these models to outperform linear methods like PCA highlights the efficiency of non-linear latent features. Future work will explore more complex datasets like CelebA and alternative generative architectures to further understand the limits of unsupervised semantic discovery.
Summary updated