Abstract
Learning controllable and generalizable representation of multivariate data with desired structural properties remains a fundamental problem in machine learning. In this paper, we present a novel framework for learning generative models with various underlying structures in the latent space. We represent the inductive bias in the form of mask variables to model the dependency structure in the graphical model and extend the theory of multivariate information bottleneck to enforce it. Our model provides a principled approach to learn a set of semantically meaningful latent factors that reflect various types of desired structures like capturing correlation or encoding invariance, while also offering the flexibility to automatically estimate the dependency structure from data. We show that our framework unifies many existing generative models and can be applied to a variety of tasks including multi-modal data modeling, algorithmic fairness, and invariant risk minimization.
Introduction
In representation learning, the objective is often to discover latent factors that are "structured"—meaning they adhere to specific constraints such as independence (disentanglement) or invariance to certain attributes. Historically, researchers have developed specialized architectures for individual tasks like multi-modal modeling, fairness, or domain generalization. However, these models frequently rely on heuristic objectives, lacking a unified, principled method to explicitly represent and enforce inductive biases within a generative framework. This paper addresses this gap by establishing a mathematical connection between graphical models and information theory.
Methodology
The proposed framework treats dependency structures as learnable components through several key innovations:
Explicit Dependency Representation
The model utilizes binary mask matrices ($M^p$ and $M^q$) to define the contributions of latent factors to observed variables. This approach transforms the latent space into a Bayesian Network where the structure itself is subject to optimization. This flexibility allows the model to represent a wide array of dependency structures, from joint to private latent factors.
Information-Theoretic Objective
By extending Multivariate Information Bottleneck (MIB) theory, the authors derive a generalized Lagrangian objective. This objective balances data reconstruction (distribution consistency) with structural adherence in the latent space (structural regularization).
Tractable Inference and Structure Learning
To accommodate missing data, particularly in multi-modal contexts, a "weighted product-of-experts" inference model is employed. Furthermore, the framework leverages the Gumbel-softmax trick to make discrete structure variables differentiable, enabling the model to learn the optimal dependency graph directly from data.
Key Insights
A significant contribution of this work is the unification of various generative models. Frameworks such as VAE, $\beta$-VAE, GAN, InfoGAN, and JMVAE can be viewed as special cases of this model by adjusting structural constraints and Lagrangian multipliers. Additionally, the use of mask variables as "gates" maintains mathematical tractability for high-dimensional data while allowing for complex, non-linear modeling.
Experimental Results
The framework was evaluated across three distinct domains:
- Multi-modal Modeling: The model successfully disentangled "shared" latent factors (e.g., digit identity) from "private" factors (e.g., domain-specific style), enabling high-quality cross-domain generation.
- Algorithmic Fairness: On UCI Adult and German datasets, the model achieved competitive accuracy while significantly reducing the Difference of Equal Opportunity (DEO). On the Adult dataset, it reached a DEO of 0.01 with 79-83% accuracy.
- OOD Generalization: On Colored-MNIST, the framework performed comparably to Invariant Risk Minimization (IRM), even in semi-supervised settings with only 50% labeled data.
Discussion and Limitations
While the framework offers a powerful unifying perspective, it faces challenges. The inference model's reliance on Gaussian assumptions for the product-of-experts may limit its effectiveness for highly complex multi-modal posteriors. Additionally, the search space for optimal Bayesian network structures is super-exponential, posing scalability issues for models with hundreds of variables. Finally, the introduction of multiple Lagrangian multipliers ($\alpha, \beta, \gamma$) necessitates careful hyperparameter tuning to balance reconstruction and structural integrity.
Summary updated