Abstract
Generative Adversarial Networks (GANs) are modern methods to learn the underlying distribution of a data set. GANs have been widely used in sample synthesis, de-noising, domain transfer, etc. GANs, however, are designed in a model-free fashion where no additional information about the underlying distribution is available. In many applications, however, practitioners have access to the underlying independence graph of the variables, either as a Bayesian network or a Markov Random Field (MRF). We ask: how can one use this additional information in designing model-based GANs? In this paper, we provide theoretical foundations to answer this question by studying subadditivity properties of probability divergences, which establish upper bounds on the distance between two high-dimensional distributions by the sum of distances between their marginals over (local) neighborhoods of the graphical structure of the Bayes-net or the MRF. We prove that several popular probability divergences satisfy some notion of subadditivity under mild conditions. These results lead to a principled design of a model-based GAN that uses a set of simple discriminators on the neighborhoods of the Bayes-net/MRF, rather than a giant discriminator on the entire network, providing significant statistical and computational benefits. Our experiments on synthetic and real-world datasets demonstrate the benefits of our principled design of model-based GANs.
Introduction: Beyond Model-Free GANs
Standard Generative Adversarial Networks (GANs) are typically "model-free," attempting to learn complex high-dimensional data distributions from scratch. While versatile, this approach faces a significant statistical hurdle known as the "curse of dimensionality." Training a single, massive discriminator to distinguish between real and fake high-dimensional samples is often computationally unstable and requires vast amounts of data.
However, in fields ranging from time-series analysis to medical diagnostics, we often possess prior knowledge regarding the structural relationships between variables. This structure is typically represented as a Conditional Independence Graph, such as a Bayesian Network or a Markov Random Field (MRF). This paper explores how to leverage this "skeleton" of data to design more efficient, model-based GANs.
Theoretical Foundation: Subadditivity of Divergences
The core innovation of this work is the application of subadditivity. The researchers prove that the distance between two joint distributions ($P$ and $Q$) can be upper-bounded by the sum of the distances between their local marginals—specifically, the neighborhoods defined by the data's underlying graph structure.
By proving that popular divergences (like Jensen-Shannon, Wasserstein, and various $f$-divergences) satisfy generalized subadditivity, the authors justify a "divide and conquer" architecture. Instead of one global discriminator, the Bayes-net/MRF GAN utilizes multiple "small" discriminators. Each local discriminator focuses only on a specific node and its parents (in a Bayes-net) or a maximal clique (in an MRF). The generator's objective is then to satisfy this committee of local experts by minimizing the sum of these local divergence estimates.
Experimental Evaluation and the "Goldilocks Zone"
The proposed framework was tested on several benchmarks, including the synthetic Ball Throwing Trajectories (a 15-step time-series governed by physics) and real-world Bayesian networks like EARTHQUAKE and CHILD.
A key finding is the existence of a "U-shaped" trade-off regarding discriminator size. If a local discriminator's view is too narrow (e.g., looking at only 2 points of a trajectory when the physics requires 3), it fails to capture the essential dynamics. Conversely, if the view is too wide (approaching the global standard GAN), the statistical burden leads to noisy training and slower convergence.
As shown in the trajectory results, the model-based approach with the correct "localization width" (Width 3) produces samples that closely mirror the ground truth. In contrast, standard GANs often "hallucinate" dependencies or produce physically implausible samples. Quantitatively, the Bayes-net GAN consistently outperformed standard models in metrics like Relative BIC and Detection AUC, while also allowing for more accurate recovery of the true causal graph from generated data.
Insights and Limitations
- Causal Integrity: By enforcing local constraints, the model respects the conditional independence of the graph, leading to logically consistent samples.
- Divergence Choice: Interestingly, the authors found that the 2-Wasserstein distance does not always satisfy subadditivity for Gaussians, suggesting that the choice of loss function is critical in model-based design.
- Limitations: The primary requirement is that the graph structure must be known beforehand. Future work may involve combining this framework with structure learning to discover the graph from data before training the GAN.
In conclusion, by shifting from a single giant discriminator to a structured set of local ones, model-based GANs offer a principled path toward faster convergence and higher-quality generative modeling in structured domains.
Summary updated