The Transitive Information Theory and its Application to Deep Generative Models

Trung Ngo Trong, Najwa Laabid, Ville Hautamäki, Merja HeinäniemiarXiv 2022 · DOI · OpenReview

Abstract

Variational Autoencoders (VAEs) often struggle with the rate-distortion trade-off, where improving image reconstruction quality typically comes at the cost of latent space disentanglement. This paper introduces Transitive Information Theory, a mathematical framework that leverages a third "bridge" variable to maximize mutual information between data and latents. We apply this theory to SemafoVAE, a semi-supervised architecture that utilizes a controllable prior to achieve high-fidelity generation and superior factor separation. Evaluations on MNIST, Fashion-MNIST, and Shapes3D demonstrate that SemafoVAE outperforms existing methods in both sharpness and disentanglement metrics.

Introduction: The Rate-Distortion Paradox

Variational Autoencoders are foundational in generative modeling, yet they are plagued by a persistent trade-off. If a model is optimized for highly realistic, sharp images, it often suffers from "posterior collapse," where the latent variables are ignored. Conversely, forcing the model to learn independent, disentangled factors (e.g., rotation, color) usually results in blurry reconstructions. Standard VAEs rely on a simple noise prior, which lacks the structure necessary to enforce a hierarchy or provide explicit control over generated outputs.

Transitive Information Theory

The core theoretical contribution is a theorem stating that the mutual information between two variables—such as an image $x$ and its latent code $z$—can be lower-bounded by their joint relationship to a third variable $y$ (representing ground truth factors). In this framework, $y$ acts as an information bridge, allowing the model to bypass the traditional VAE bottleneck.

Venn diagrams illustrating the transitive relationship between data $H(X)$, latent $H(Z)$, and factors $H(Y)$.

By maximizing the information flow through this bridge, the model can maintain a rich latent representation without sacrificing the fine-grained details required for high-quality image generation.

SemafoVAE: A Dual-VAE Architecture

Inspired by the "semaphore" concept, the authors propose SemafoVAE, a semi-supervised model consisting of two jointly learned components:

  1. Reconstruction VAE: Maps images to latent codes to ensure high-fidelity reconstruction.
  2. Controller VAE: Learns a "controllable prior." Instead of sampling from isotropic Gaussian noise, the model learns to navigate the latent space using a small amount of labeled data.
Graphical models (a-c) and the dual-VAE architecture (d) of SemafoVAE, featuring a controllable prior and reconstruction pathway.

This architecture introduces an inductive bias into the prior, enabling the learning of a hierarchy of disentangled representations while keeping the generator expressive.

Key Insights and Results

The authors reframe the VAE objective as a game between an encoder (compression) and a decoder (recovery). They observe that "active unit limits" are often architecture-dependent; for instance, MNIST models consistently converge to approximately 25 active latent units regardless of the total dimensions provided.

SemafoVAE and its hierarchical variant, SemafoHVAE, were tested on MNIST, Fashion-MNIST, and Shapes3D. Even with extremely low labeling rates (as low as 0.004%), the model showed significant improvements:

  • Generation Quality: On Shapes3D, SemafoHVAE achieved a Fréchet Inception Distance (FID) of 62.36, outperforming the vanilla VAE (74.57) and the M2 baseline (81.63).
  • Disentanglement: The model achieved a DCI score of 82.62 on Shapes3D, compared to 64.82 for a standard VAE, indicating much cleaner separation of generative factors.
  • Visual Fidelity: Qualitative results confirm that SemafoVAE generates sharper, more distinct images from its prior, avoiding the "ghostly" averages typical of previous methods.

Conclusion and Future Work

SemafoVAE successfully bypasses the rate-distortion trade-off by utilizing Transitive Information Theory to structure the latent space. While highly effective on benchmark datasets, future research will focus on scaling this logic to high-resolution, real-world images and exploring fully unsupervised versions of the transitive framework.

Summary updated

Abstract

Paradoxically, a Variational Autoencoder (VAE) could be pushed in two opposite directions, utilizing powerful decoder model for generating realistic images but collapsing the learned representation, or increasing regularization coefficient for disentangling representation but ultimately generating blurry examples. Existing methods narrow the issues to the rate-distortion trade-off between compression and reconstruction. We argue that a good reconstruction model does learn high capacity latents that encode more details, however, its use is hindered by two major issues: the prior is random noise which is completely detached from the posterior and allow no controllability in the generation; mean-field variational inference doesn't enforce hierarchy structure which makes the task of recombining those units into plausible novel output infeasible. As a result, we develop a system that learns a hierarchy of disentangled representation together with a mechanism for recombining the learned representation for generalization. This is achieved by introducing a minimal amount of inductive bias to learn controllable prior for the VAE. The idea is supported by here developed transitive information theory, that is, the mutual information between two target variables could alternately be maximized through the mutual information to the third variable, thus bypassing the rate-distortion bottleneck in VAE design. In particular, we show that our model, named SemafoVAE (inspired by the similar concept in computer science), could generate high-quality examples in a controllable manner, perform smooth traversals of the disentangled factors and intervention at a different level of representation hierarchy.
BibTeX
@misc{trong2022transitive,
  author = {Trung Ngo Trong and Najwa Laabid and Ville Hautamäki and Merja Heinäniemi},
  title = {{The Transitive Information Theory and its Application to Deep Generative Models}},
  year = {2022},
  eprint = {2203.05074},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.05074},
}
Figure