A Deep Generative Approach to Conditional Sampling

Xingyu Zhou, Yuling Jiao, Jin Liu, Jian Huang

This paper presents a theoretically grounded deep learning framework for conditional sampling, moving beyond simple mean-based regression to capture the full complexity of conditional distributions.

TL;DR

The authors introduce the Generative Conditional Distribution Sampler (GCDS), a deep learning framework that learns to draw samples from complex conditional distributions $P(Y|X)$. By leveraging the "noise-outsourcing lemma" and matching joint distributions via the variational form of KL divergence, GCDS handles high-dimensional data and provides a rare proof of theoretical consistency for deep generative models.

What Problem Does This Paper Address?

In many real-world scenarios, knowing the "average" outcome (the conditional mean) isn't enough. If a distribution is multimodal (has multiple peaks) or highly skewed, the mean might actually be an unlikely outcome. To fully understand the relationship between a predictor $X$ and a response $Y$, we need the entire conditional distribution $P(Y|X)$.

Traditional methods for Conditional Density Estimation (CDE), such as kernel smoothing or nearest neighbors, suffer from the "curse of dimensionality." They break down when $X$ or $Y$ are high-dimensional (like images) or when the relationship is highly non-linear. While Conditional GANs (cGANs) have been used for this, they often lack theoretical guarantees and can struggle with continuous conditioning variables.

Methods

The core of GCDS is the Noise-Outsourcing Lemma. This probability theory result states that any conditional distribution can be represented by a function $G(\eta, X)$, where $\eta$ is a random noise variable from a simple reference distribution (like a standard Normal).

The authors propose a two-network architecture:

  1. The Conditional Generator ($G$): A neural network that takes a predictor $x$ and noise $\eta$ to produce a sample $y$.
  2. The Discriminator ($D$): A network that acts as a "critic," helping the generator match the target distribution.

The Clever Trick: Instead of trying to match the conditional distribution directly (which is difficult when $X$ is continuous), the authors prove that matching the joint distribution $(X, G(\eta, X))$ to the true joint distribution $(X, Y)$ is mathematically equivalent. They achieve this using the variational (dual) form of the Kullback-Leibler (KL) divergence, resulting in a minimax optimization problem similar to a GAN but specifically designed for conditional sampling.

Datasets

The authors validated GCDS across a diverse range of data types:

  • Simulated Models (M1–M4): Four synthetic models featuring additive noise, heteroscedasticity (variance changing with $X$), non-Gaussian errors, and multimodal (mixture) distributions.
  • Abalone Dataset (UCI): A tabular dataset used to predict the number of rings (age) of abalones based on physical measurements.
  • MNIST: The classic handwritten digits dataset, used for generating images from labels and reconstructing missing parts of images (inpainting).
  • STL-10: A dataset of color images ($96 \times 96 \times 3$) used to demonstrate high-dimensional image reconstruction.

Key Insights and Ideas

  • Sampling vs. Density Estimation: The authors argue that for high-dimensional data, it is more practical to learn a sampler than to estimate a density function. Once you can sample, you can estimate any summary statistic (mean, variance, quantiles) via Monte Carlo methods.
  • Theoretical Consistency: One of the paper's strongest contributions is a proof of weak convergence. They show that under mild conditions, as the sample size $n$ and the neural network size increase, the distribution produced by the generator converges to the true underlying conditional distribution. This provides a rigorous foundation often missing in deep learning literature.
  • Unified Formulation: GCDS treats regression, classification, and image generation under a single mathematical umbrella. It handles continuous and discrete predictors/responses natively without changing the loss function.

Key Results

  • Superior Accuracy: In simulated studies, GCDS consistently achieved lower Mean Squared Error (MSE) for both conditional means and standard deviations compared to established methods like CKDE and FlexCode.
  • Multimodal Mastery: GCDS successfully captured bimodal distributions (Model M4) where traditional regression would fail by simply predicting the (unlikely) midpoint between two peaks.
  • Uncertainty Quantification: On the Abalone dataset, GCDS produced 90% prediction intervals that achieved an actual coverage of 89.71%, demonstrating highly reliable uncertainty estimation.
  • High-Dimensional Scaling: The model successfully performed image inpainting on STL-10, where the response $Y$ was a 6,912-dimensional vector (the corrupted pixels), a task far beyond the reach of classical CDE methods.

Limitations and Future Work

  • Convergence Rates: While the authors proved consistency (that the model eventually reaches the truth), they noted that deriving the specific rate of convergence (how much data is needed for a specific accuracy) remains an open challenge.
  • Density Function Recovery: GCDS provides samples, not a functional form of the density. If an explicit formula for the density is required, one must apply additional techniques (like kernel smoothing) to the generated samples.
  • Complex Inpainting: The authors noted that while GCDS works for simple image reconstruction, more work is needed to handle multiple or irregularly shaped corrupted areas in complex images.

Summary updated

Abstract

We propose a deep generative approach to sampling from a conditional distribution based on a unified formulation of conditional distribution and generalized nonparametric regression function using the noise-outsourcing lemma. The proposed approach aims at learning a conditional generator, so that a random sample from the target conditional distribution can be obtained by transforming a sample drawn from a reference distribution. The conditional generator is estimated nonparametrically with neural networks by matching appropriate joint distributions using the Kullback-Liebler divergence. An appealing aspect of our method is that it allows either of or both the predictor and the response to be high-dimensional and can handle both continuous and discrete type predictors and responses. We show that the proposed method is consistent in the sense that the conditional generator converges in distribution to the underlying conditional distribution under mild conditions. Our numerical experiments with simulated and benchmark image data validate the proposed method and demonstrate that it outperforms several existing conditional density estimation methods. Supplementary materials for this article are available online.
BibTeX
@article{zhou2021deep,
  author = {Xingyu Zhou and Yuling Jiao and Jin Liu and Jian Huang},
  title = {{A Deep Generative Approach to Conditional Sampling}},
  journal = {Journal of the American Statistical Association},
  year = {2021},
  doi = {10.1080/01621459.2021.2016424},
}
Figure