Abstract
Generative Adversarial Networks (GANs) are modern methods to learn the underlying distribution of a data set. GANs have been widely used in sample synthesis, de-noising, domain transfer, etc. GANs, however, are designed in a model-free fashion where no additional information about the underlying distribution is available. In many applications, however, practitioners have access to the underlying independence graph of the variables, either as a Bayesian network or a Markov Random Field (MRF). We ask: how can one use this additional information in designing model-based GANs? In this paper, we provide theoretical foundations to answer this question by studying subadditivity properties of probability divergences, which establish upper bounds on the distance between two high-dimensional distributions by the sum of distances between their marginals over (local) neighborhoods of the graphical structure of the Bayes-net or the MRF. We prove that several popular probability divergences satisfy some notion of subadditivity under mild conditions. These results lead to a principled design of a model-based GAN that uses a set of simple discriminators on the neighborhoods of the Bayes-net/MRF, rather than a giant discriminator on the entire network, providing significant statistical and computational benefits. Our experiments on synthetic and real-world datasets demonstrate the benefits of our principled design of model-based GANs.
Introduction
Generative Adversarial Networks (GANs) are typically "model-free," meaning they attempt to learn complex data distributions without any prior knowledge of how variables relate to one another. While flexible, this approach struggles in high dimensions. Training a "giant" global discriminator to distinguish real from fake data requires an enormous amount of samples—often exponentially more as the number of variables increases.
In many real-world scenarios, however, we have access to the underlying "independence graph" (which variables directly influence others). This research addresses how to mathematically and architecturally incorporate this structural information into GAN design to break the curse of dimensionality.
Methods: Subadditivity and Local Discriminators
The core contribution of this work is the study of subadditivity properties of probability divergences. The authors prove that for many popular divergences—including Jensen-Shannon, Kullback-Leibler, and Wasserstein distance—the distance between two high-dimensional distributions can be upper-bounded by the sum of distances between their local marginals (neighborhoods in the graph).
Based on this theory, they propose a "model-based" GAN architecture. Instead of one global discriminator, they use a set of "small" discriminators, each focused on a local neighborhood defined by the data's Bayesian Network or Markov Random Field (MRF). This approach is further justified by Theorem 8, which proves that "neural distances" (IPMs defined by ReLU neural networks) satisfy generalized subadditivity, providing a solid theoretical basis for using standard neural architectures as local discriminators.
Experimental Results
The researchers evaluated their approach on three main benchmarks: a synthetic 15-step ball-throwing trajectory dataset and two real-world Bayesian Networks (EARTHQUAKE and CHILD).
Key Insights
- The Power of Locality: If local conditional distributions are matched perfectly, the global distribution follows. This allows for significant statistical efficiency.
- The "U-Shaped" Trade-off: There is an optimal "localization width" for discriminators. If discriminators are too local, they miss critical dependencies; if they are too global, they become too complex to train effectively.
- Causal Integrity: Model-based GANs are superior at capturing the actual causal structure of the data, whereas standard GANs often "hallucinate" dependencies between variables that should be independent.
Visual and Quantitative Performance
In the ball-throwing experiment, the model-based GAN (width 3) produced smooth, physically plausible trajectories. In contrast, the standard GAN produced noisy, jittery curves that failed to capture the underlying physics.
Quantitatively, on the EARTHQUAKE dataset, the model-based GAN improved the "Energy Statistics" (a measure of distributional similarity) from 1.72 to 0.24—a nearly 7x improvement. Additionally, the proposed GANs converged faster and were more computationally efficient because the local discriminators are significantly smaller and easier to optimize than a single global one.
Conclusion
This framework provides a principled design for model-based GANs that leverage known independence structures. While the current method assumes the independence graph is known, it provides the foundation for future work in structure learning, where GANs might simultaneously learn the independence graph and the data distribution.
Summary updated