Adversarial sampling of unknown and high-dimensional conditional distributions

Malik Hassanaly, Andrew Glaws, Karen Stengel, Ryan N. King

TL;DR

This paper introduces a novel adversarial approach using Generative Adversarial Networks (GANs) to sample from complex, high-dimensional conditional distributions. By incorporating estimated conditional moments into the GAN's loss function, the method effectively improves sample diversity and quality, particularly for challenging problems like deconvolution of turbulent flow fields.

What Problem Does This Paper Address?

Many engineering and scientific problems require predicting the behavior of systems with many degrees of freedom, often conditioned on partial observations. For instance, in weather forecasting, predicting future states requires sampling from a high-dimensional probability distribution of atmospheric conditions, given current measurements. When these distributions are unknown and high-dimensional, traditional sampling methods like Markov Chain Monte Carlo (MCMC) struggle. Existing data-driven methods, like Generative Adversarial Networks (GANs), can sample from distributions, but face challenges when dealing with conditional distributions, especially when the conditioning variables are continuous. A key issue is "mode collapse," where the GAN fails to capture the full diversity of the target distribution. This paper tackles the problem of generating diverse and accurate samples from unknown, high-dimensional conditional distributions, a crucial task for uncertainty quantification, simulation, and inference in various scientific domains.

Methods

The core of the proposed method is an adversarial approach leveraging Conditional Generative Adversarial Networks (CGANs). The novelty lies in augmenting the standard CGAN training with a "diversity loss" that uses a priori estimates of the conditional moments (mean and variance) of the target distribution.

Here's a breakdown:

  • Conditional GANs (CGANs): These networks are adapted to generate samples conditioned on specific inputs (e.g., low-resolution data).
  • Conditional Moment Estimation: The paper proposes two methods to estimate the conditional moments (specifically, the first and second moments, related to mean and variance) of the target distribution from the training data:
  • Neural Network-Assisted Estimation: A neural network is trained to approximate the conditional moments.
  • Stochastic Estimation: A functional basis (polynomial expansion) is used to approximate the conditional moments.
  • Diversity Loss: This new loss term encourages the generated samples to match the estimated conditional moments. It's inspired by metrics like the Fréchet Inception Distance (FID) but adapted for continuous conditioning variables. This loss helps combat mode collapse and ensures the generated samples span the correct distribution.
  • Content Loss: An additional loss term is used to ensure that the generated samples satisfy the conditioning constraint (e.g., that a deconvolved field, when filtered, matches the low-resolution input).

Datasets

The primary dataset used for training and evaluation is from the National Renewable Energy Laboratory's Wind Integration National Database (WIND) Toolkit. This dataset contains easterly and northerly wind speeds at 100m height over the continental United States for the years 2007-2013.

  • High-Resolution (HR) Dataset: Approximately 38,000 random snapshots of 100x100 pixels (10 km/pixel resolution).
  • Low-Resolution (LR) Dataset: Generated by applying a box filter to the HR data, resulting in 10x10 pixels (100 km/pixel resolution).

The paper uses this dataset to demonstrate the deconvolution of turbulent wind velocity fields.

Key Insights and Ideas

  • Conditional Moments as a Diversity Regularizer: The most significant insight is that explicitly incorporating estimates of conditional moments into the GAN's loss function is a powerful way to guide the generator towards producing diverse samples that accurately reflect the target conditional distribution, especially when conditioning variables are continuous.
  • Bridging the Gap in Conditional GANs: The paper provides a practical solution to a known limitation of GANs in handling continuous conditioning variables, where traditional methods struggle due to sparse data.
  • Flexibility in Moment Estimation: The method is agnostic to the specific technique used for estimating conditional moments, allowing for the use of either neural networks or functional expansions, offering flexibility based on problem specifics.
  • Deconvolution as a Testbed: The deconvolution of turbulent wind fields serves as a challenging and relevant benchmark, demonstrating the method's ability to handle complex, high-dimensional data with strong spatial correlations.

Key Results

The proposed method, particularly when using the stochastic estimation of conditional moments, demonstrated significant improvements in sample diversity compared to other GAN-based approaches like DSGAN and Generator Similarity.

  • Quality: GAN-based approaches, including the proposed method, generated higher fidelity samples with sharper gradients compared to traditional inverse filter approximations (ADM, Taylor expansion). The quality of samples generated by the proposed method was only marginally decreased compared to state-of-the-art methods, while significantly increasing diversity.
  • Diversity: The diversity metric (Eq. 12) showed that the proposed methods (both NN-assisted and stochastic estimation) achieved better diversity than DSGAN and Generator Similarity. Even suboptimal estimates of conditional moments still yielded better diversity than these baseline GANs.
  • Consistency: All methods, including the proposed ones, produced generated fields that were consistent with their convolved counterparts, as quantified by a content loss metric (Eq. 15).
  • Efficiency: The new diversity loss led to faster training than DSGAN and Generator Similarity (approximately 1.5 times faster per epoch).

Limitations and Future Work

  • Covariance Matrix Assumption: The paper assumes a diagonal covariance matrix for the unresolved data. While the discriminator compensated for this, the authors suggest that a diagonal-by-block covariance matrix could provide a finer description of the distribution.
  • Optimal Moment Estimation: There's no guarantee that the estimated conditional moments are truly optimal. The authors suggest that increasing the complexity of the moment estimator is the way to minimize estimation error, which can be costly.
  • Transferability of Moments: Future work could investigate how transferable learned moments are across different but related datasets or surrogate models.
  • On-the-fly Moment Computation: Instead of storing precomputed moments, computing them on-the-fly using polynomial functions or external neural networks could improve memory efficiency.
  • Adaptive Architectures: Exploring different neural network architectures or functional bases for different snapshots, rather than a single model for all, could be beneficial.
  • Statistical Uncertainty: Incorporating the statistical uncertainty of moment estimates into the loss function could prevent the generation of diverse samples solely due to estimation errors.

Summary updated

Abstract

Many engineering problems require the prediction of realization-to-realization variability or a refined description of modeled quantities. In that case, it is necessary to sample elements from unknown high-dimensional spaces with possibly millions of degrees of freedom. While there exist methods able to sample elements from probability density functions (PDF) with known shapes, several approximations need to be made when the distribution is unknown. In this paper the sampling method, as well as the inference of the underlying distribution, are both handled with a data-driven method known as generative adversarial networks (GAN), which trains two competing neural networks to produce a network that can effectively generate samples from the training set distribution. In practice, it is often necessary to draw samples from conditional distributions. When the conditional variables are continuous, only one (if any) data point corresponding to a particular value of a conditioning variable may be available, which is not sufficient to estimate the conditional distribution. This work handles this problem using an a priori estimation of the conditional moments of a PDF. Two approaches, stochastic estimation, and an external neural network are compared here for computing these moments; however, any preferred method can be used. The algorithm is demonstrated in the case of the deconvolution of a filtered turbulent flow field. It is shown that all the versions of the proposed algorithm effectively sample the target conditional distribution with minimal impact on the quality of the samples compared to state-of-the-art methods. Additionally, the procedure can be used as a metric for the diversity of samples generated by a conditional GAN (cGAN) conditioned with continuous variables.
BibTeX
@article{hassanaly2022adversarial,
  author = {Malik Hassanaly and Andrew Glaws and Karen Stengel and Ryan N. King},
  title = {{Adversarial sampling of unknown and high-dimensional conditional distributions}},
  journal = {Journal of Computational Physics},
  year = {2022},
  doi = {10.1016/j.jcp.2021.110853},
}
Figure