Bridging Disentanglement with Independence and Conditional Independence via Mutual Information for Representation Learning

Xiaojiang Yang, Wendong Bi, Yu Cheng, Junchi YanarXiv 2019 · OpenReview

TL;DR

This paper proposes a new theoretical framework for disentangled representation learning by introducing a novel assumption based on mutual information. They demonstrate that a key inductive bias, the factorized approximate posterior in VAEs and InfoGANs, is crucial for achieving disentanglement by enforcing conditional independence between latent factors and the data.

What Problem Does This Paper Address?

Disentangled representation learning aims to learn latent representations where each dimension (or factor) corresponds to an independent, interpretable factor of variation in the data. While many existing methods rely on the assumption that these latent factors should be marginally independent (i.e., their joint distribution is the product of their marginals), this paper argues that this assumption is insufficient. The core problem is that marginal independence alone doesn't guarantee that each latent factor truly captures a distinct, meaningful aspect of the data, independent of other factors. The authors propose to address this by characterizing disentanglement through the relationship between the latent factors and the observable data, specifically using mutual information.

Methods

The paper introduces a new assumption for disentangled representations: the mutual information between each latent factor and the data should be invariant with respect to the other latent factors. This is mathematically expressed as $I(z_j; x | \{z_i\}_{i \in S_{-j}}) = I(z_j; x)$, where $z_j$ is a latent factor, $x$ is the data, and $\{z_i\}_{i \in S_{-j}}$ are all other latent factors.

They then show that this assumption, combined with the existing marginal independence assumption, implies conditional independence of the latent factors given the data: $q(z|x) = \prod_j q(z_j|x)$. This conditional independence is a crucial insight. The authors argue that this property is often implicitly enforced in popular models like Variational Autoencoders (VAEs) and InfoGAN through their use of a factorized approximate posterior in the encoder. This factorized posterior acts as an inductive bias that is essential for disentanglement. To test their hypothesis, they experimentally violate this conditional independence by using correlated approximate posteriors and observe a decline in disentanglement.

Datasets

The paper uses the following datasets for evaluation:

  • dSprites: A dataset of 737,280 grayscale images of simple shapes, generated from five independent latent factors (shape, scale, rotation, x-position, y-position).
  • SmallNORB: A dataset containing 24,300 image pairs of 3D toys from different viewpoints, with variations in lighting, pose, and scale.
  • Cars3D: A dataset of 199 colorful 3D car models, varying in viewpoint and other factors.
  • MNIST: A dataset of handwritten digits, used for qualitative evaluation with InfoGAN.

Key Insights and Ideas

  • Marginal Independence is Not Enough: The paper strongly argues that simply enforcing independence between latent factors is insufficient for learning truly disentangled representations. The relationship with the data matters.
  • Mutual Information as a Disentanglement Metric: The proposed assumption uses mutual information to quantify how much information each latent factor carries about the data, and crucially, how this information is preserved even when conditioning on other latent factors.
  • Conditional Independence is Key: The core theoretical contribution is bridging the mutual information assumption with conditional independence of latent factors given the data. This is a more direct link to how latent factors should relate to the observable data.
  • Factorized Posterior as an Inductive Bias: The paper identifies the factorized approximate posterior in VAEs and InfoGANs not just as a convenient choice, but as a critical inductive bias that enforces the necessary conditional independence for disentanglement.
  • Violating Conditional Independence Hurts Disentanglement: The experimental results provide strong evidence that breaking this conditional independence (by using correlated posteriors) leads to a significant degradation in disentanglement scores.

Key Results

  • VAE Experiments:
    • When using correlated approximate posteriors (varying $\sigma$ from 0 to 0.9) in vanilla VAE, FactorVAE, and TCVAE, the Mutual Information Gap (MIG) scores consistently declined across dSprites, SmallNORB, and Cars3D datasets. This decline was observed for both 10 and 20 factors.
  • The impact on reconstruction error, KL term, and total correlation was generally minor, suggesting that the decline in disentanglement was not due to general model instability but specifically to the violation of conditional independence.
  • InfoGAN Experiments:
    • Qualitative results on MNIST showed that when using correlated approximate posteriors, traversing latent factors resulted in less interpretable and disentangled image variations compared to using factorized posteriors. For example, factors that clearly controlled rotation and width in the $\sigma=0$ case became mixed and less interpretable at $\sigma=0.9$.
  • Loss curves for InfoGAN training showed that both correlated and uncorrelated posteriors converged well, indicating that the factorized posterior's benefit is in disentanglement quality, not training stability.

Limitations and Future Work

  • Deterministic Case: The paper notes that mutual information $I(z; x)$ can be infinite when the relationship between $z$ and $x$ is deterministic. While this is addressed by considering stochastic representations, it's a theoretical edge case.
  • Future Work: The authors suggest exploring the relationship between their proposed necessary conditions for disentanglement and the sufficient conditions derived from nonlinear ICA theory. They also mention the importance of further investigating the inductive bias of the factorized approximate posterior.

Summary updated

Abstract

Most existing works on disentangled representation learning are solely built upon an marginal independence assumption: all factors in disentangled representations should be statistically independent. This assumption is necessary but definitely not sufficient for the disentangled representations without additional inductive biases in the modeling process, which is shown theoretically in recent studies. We argue in this work that disentangled representations should be characterized by their relation with observable data. In particular, we formulate such a relation through the concept of mutual information: the mutual information between each factor of the disentangled representations and data should be invariant conditioned on values of the other factors. Together with the widely accepted independence assumption, we further bridge it with the conditional independence of factors in representations conditioned on data. Moreover, we note that conditional independence of latent variables has been imposed on most VAE-type models and InfoGAN due to the artificial choice of factorized approximate posterior $q(\rvz|\rvx)$ in the encoders. Such an arrangement of encoders introduces a crucial inductive bias for disentangled representations. To demonstrate the importance of our proposed assumption and the related inductive bias, we show in experiments that violating the assumption leads to decline of disentanglement among factors in the learned representations.
BibTeX
@misc{yang2019bridging,
  author = {Xiaojiang Yang and Wendong Bi and Yu Cheng and Junchi Yan},
  title = {{Bridging Disentanglement with Independence and Conditional Independence via Mutual Information for Representation Learning}},
  year = {2019},
  eprint = {1911.10922},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.10922},
}
Figure