Learning Structured Latent Factors from Dependent Data:A Generative Model Framework from Information-Theoretic Perspective

Ruixiang Zhang, Masanori Koyama, Katsuhiko Ishiguro

In the quest for artificial intelligence that truly understands the world, learning "disentangled" or "structured" representations is a holy grail. We want models that don't just memorize data but decompose it into meaningful factors—like separating the identity of a person from their pose in a photo.

While many models like $\beta$-VAEs attempt this, they often rely on heuristic tricks tailored to specific tasks. In their ICML 2020 paper, Zhang et al. propose a principled, information-theoretic framework that unifies these disparate approaches into a single, flexible architecture capable of learning complex dependency structures from data.

The Challenge of Dependent Data

Most current generative models are highly specialized. A model designed for disentanglement might fail at multi-modal fusion, and a model built for fairness often requires a completely different architecture. This lack of a unified language for "inductive bias"—the set of assumptions a model makes about the data—makes it difficult to transfer insights between domains.

Furthermore, many models struggle with "dependent data," where different parts of the input (like an image and its corresponding text caption) share some information but also have unique, private variations. Existing methods often rely on heuristics to decide which latent variables should be shared and which should be private.

A Unified Methodological Framework

The core of this work is the use of explicit dependency structure representation to model how latent factors interact.

  1. Mask Variables: The authors introduce binary mask matrices that act as "gates" in the graphical model. These masks define which latent factors contribute to which observed variables during generation, and which observed variables inform which latent factors during inference.
  2. Information-Theoretic Objective: They generalize the Multivariate Information Bottleneck (MIB) theory. The training objective minimizes the divergence between the learned distribution and the desired prior, while strictly enforcing structural constraints defined by Bayesian networks.
  3. Tractable Inference via Product-of-Experts: To handle the massive number of possible dependency structures, they use a weighted Product-of-Experts (PoE) approach. This allows the model to handle missing data naturally: if a modality is missing, its "expert" simply doesn't contribute to the final latent representation.
  4. Structure Learning: Instead of manually hard-coding the graph, the framework can learn the optimal masks using gradient-based optimization via the Gumbel-softmax trick.
Bayesian networks for multi-modal settings, illustrating shared and private latent factors.

Key Insights

  • The Great Unifier: Perhaps the most impressive feat is demonstrating that many famous models—including VAE, GAN, InfoGAN, $\beta$-VAE, and JMVAE—are actually just special cases of this framework with specific mask configurations and loss weights.
  • Principled Sharing: By utilizing specific cross and private structures (as seen in the figure above), the model mathematically separates "common" latent factors (shared across modalities) from "private" ones (unique to one modality).
  • Fairness as Structure: The authors show that algorithmic "fairness" can be framed as a structural constraint. By forcing the latent representation to be independent of sensitive attributes (like race or gender) within the graphical model, they achieve fair representations without relying on complex ad-hoc penalties.

Key Results

The authors evaluated their framework on several benchmarks across three distinct domains: multi-modal generation, algorithmic fairness, and out-of-distribution (OOD) generalization.

  • Multi-modal Generation: The model successfully performed cross-domain generation. For example, it can generate an MNIST digit from an SVHN input, demonstrating that it correctly captured the shared "digit identity" in the latent space while ignoring domain-specific style.
Cross-domain generation results showing successful translation from SVHN to MNIST.
  • Algorithmic Fairness: On the Adult and German credit datasets, the framework achieved state-of-the-art results in "Difference of Equal Opportunity" (DEO). It significantly reduced bias while maintaining high classification accuracy. On the Adult dataset, it achieved a DEO of 0.01-0.02, vastly outperforming standard Neural Networks (0.14).
  • OOD Generalization: On the Colored-MNIST dataset—which tests if a model can ignore spurious correlations (like digit color) to focus on the true causal factor (digit shape)—the model achieved a test accuracy of ~62%. This is comparable to specialized methods like Invariant Risk Minimization (IRM) and vastly superior to standard Empirical Risk Minimization (ERM), which crashed to ~17%.

Limitations and Future Directions

While the framework is powerful, the authors acknowledge a few areas for growth. The Lagrangian multipliers ($\beta$) that balance structural regularization and data reconstruction still require manual tuning as hyperparameters. Additionally, while gradient-based structure learning is more efficient than discrete search, scaling this approach to thousands of variables remains a computational challenge. Future work may extend this to even more complex data types and explore sophisticated mutual information estimators to further improve the training objective.

Summary updated

Abstract

Learning controllable and generalizable representation of multivariate data with desired structural properties remains a fundamental problem in machine learning. In this paper, we present a novel f...
BibTeX
@inproceedings{zhang2020learning,
  author = {Ruixiang Zhang and Masanori Koyama and Katsuhiko Ishiguro},
  title = {{Learning Structured Latent Factors from Dependent Data:A Generative Model Framework from Information-Theoretic Perspective}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=KyYzre-fImj},
}
Figure