Abstract
Variational Autoencoders (VAEs) often struggle with the rate-distortion trade-off, where improving image reconstruction quality typically comes at the cost of latent space disentanglement. This paper introduces Transitive Information Theory, a mathematical framework that leverages a third "bridge" variable to maximize mutual information between data and latents. We apply this theory to SemafoVAE, a semi-supervised architecture that utilizes a controllable prior to achieve high-fidelity generation and superior factor separation. Evaluations on MNIST, Fashion-MNIST, and Shapes3D demonstrate that SemafoVAE outperforms existing methods in both sharpness and disentanglement metrics.
Introduction: The Rate-Distortion Paradox
Variational Autoencoders are foundational in generative modeling, yet they are plagued by a persistent trade-off. If a model is optimized for highly realistic, sharp images, it often suffers from "posterior collapse," where the latent variables are ignored. Conversely, forcing the model to learn independent, disentangled factors (e.g., rotation, color) usually results in blurry reconstructions. Standard VAEs rely on a simple noise prior, which lacks the structure necessary to enforce a hierarchy or provide explicit control over generated outputs.
Transitive Information Theory
The core theoretical contribution is a theorem stating that the mutual information between two variables—such as an image $x$ and its latent code $z$—can be lower-bounded by their joint relationship to a third variable $y$ (representing ground truth factors). In this framework, $y$ acts as an information bridge, allowing the model to bypass the traditional VAE bottleneck.
By maximizing the information flow through this bridge, the model can maintain a rich latent representation without sacrificing the fine-grained details required for high-quality image generation.
SemafoVAE: A Dual-VAE Architecture
Inspired by the "semaphore" concept, the authors propose SemafoVAE, a semi-supervised model consisting of two jointly learned components:
- Reconstruction VAE: Maps images to latent codes to ensure high-fidelity reconstruction.
- Controller VAE: Learns a "controllable prior." Instead of sampling from isotropic Gaussian noise, the model learns to navigate the latent space using a small amount of labeled data.
This architecture introduces an inductive bias into the prior, enabling the learning of a hierarchy of disentangled representations while keeping the generator expressive.
Key Insights and Results
The authors reframe the VAE objective as a game between an encoder (compression) and a decoder (recovery). They observe that "active unit limits" are often architecture-dependent; for instance, MNIST models consistently converge to approximately 25 active latent units regardless of the total dimensions provided.
SemafoVAE and its hierarchical variant, SemafoHVAE, were tested on MNIST, Fashion-MNIST, and Shapes3D. Even with extremely low labeling rates (as low as 0.004%), the model showed significant improvements:
- Generation Quality: On Shapes3D, SemafoHVAE achieved a Fréchet Inception Distance (FID) of 62.36, outperforming the vanilla VAE (74.57) and the M2 baseline (81.63).
- Disentanglement: The model achieved a DCI score of 82.62 on Shapes3D, compared to 64.82 for a standard VAE, indicating much cleaner separation of generative factors.
- Visual Fidelity: Qualitative results confirm that SemafoVAE generates sharper, more distinct images from its prior, avoiding the "ghostly" averages typical of previous methods.
Conclusion and Future Work
SemafoVAE successfully bypasses the rate-distortion trade-off by utilizing Transitive Information Theory to structure the latent space. While highly effective on benchmark datasets, future research will focus on scaling this logic to high-resolution, real-world images and exploring fully unsupervised versions of the transitive framework.
Summary updated