Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data

Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelios Koutsourelakis, Paris Perdikaris

TL;DR

This paper introduces a novel deep learning framework for building surrogate models of physical systems governed by Partial Differential Equations (PDEs). Crucially, these models are trained without requiring any labeled output data, instead incorporating the governing physics directly into the learning process. This "physics-constrained" approach not only achieves comparable accuracy to traditional data-driven methods but also demonstrates superior generalization and the ability to quantify predictive uncertainty.

What Problem Does This Paper Address?

Building accurate and efficient surrogate models for complex physical systems, especially those described by PDEs, is a significant challenge. Traditionally, this involves supervised learning, where you need pairs of input parameters and corresponding output simulation results to train a model. This "labeled data" requirement can be a bottleneck, as generating high-fidelity simulation data is often computationally expensive. Furthermore, existing deep learning models, while powerful, often struggle to respect the underlying physical constraints of the system, leading to predictions that might be inaccurate or physically implausible, especially when extrapolating to unseen conditions. This paper tackles the problem of creating accurate surrogate models and quantifying their uncertainty without relying on expensive labeled simulation data, by directly embedding the physics (the PDEs themselves) into the learning objective.

Methods

The core innovation lies in incorporating the governing PDE equations directly into the loss function of the deep learning models. This allows the models to learn from input data alone, guided by the physical laws.

  • Physics-Constrained Surrogate (PCS): The authors propose using convolutional encoder-decoder neural networks. The training objective minimizes a loss function that includes terms penalizing violations of the PDE and boundary conditions. This is a departure from standard data-driven surrogates that minimize the difference between predicted and actual outputs.
  • Conditional Flow-based Generative Model: For uncertainty quantification, they employ a flow-based generative model (specifically, a conditional Glow model). This model is trained using a reverse Kullback-Leibler (KL) divergence loss between a learned conditional density and a reference Boltzmann-like density derived from the PDE's energy functional. This allows the model to learn a distribution over possible solutions, capturing predictive uncertainty without labeled data.
  • Convolutional vs. Fully-Connected Networks: The paper investigates both convolutional neural networks (CNNs) and fully-connected neural networks (FC-NNs) for parameterizing PDE solutions. They find that CNNs are significantly more effective at capturing the multiscale features of PDE solutions, particularly for flux fields, and train faster.

Datasets

The paper utilizes synthetic datasets generated from a Darcy flow problem, a common benchmark for fluid dynamics. The input fields are characterized by:

  • Gaussian Random Field (GRF): Generated using Karhunen-Loève expansion with varying parameters (KLE512, KLE128, KLE2048).
  • Warped GRF: A composition of two GRFs.
  • Channelized Field: Derived from large images.

These datasets are used for training both the physics-constrained surrogates (PCS) and data-driven surrogates (DDS) for comparison. For uncertainty quantification, a GRF KLE100 dataset is used.

Key Insights and Ideas

  • Physics as a Free Lunch for Training: The most significant insight is that the governing equations of a physical system can serve as a powerful supervisory signal, effectively replacing the need for expensive labeled output data. This opens up new avenues for surrogate modeling in data-scarce regimes.
  • CNNs Excel at Multiscale Physics: The study highlights the superior ability of CNNs, compared to FC-NNs, in capturing the complex, multiscale spatial correlations inherent in PDE solutions, especially for quantities like fluid fluxes. This is attributed to their inherent inductive bias for spatial feature extraction.
  • Uncertainty Quantification Without Labels: The use of a physics-derived reference density and reverse KL divergence allows for learning probabilistic models that quantify uncertainty, even when no labeled output data is available. This is crucial for reliable decision-making in scientific and engineering applications.
  • Generalization Beyond Training Distribution: Physics-constrained models demonstrate better generalization capabilities to out-of-distribution inputs compared to data-driven models, suggesting that learning the underlying physics leads to more robust and adaptable models.

Key Results

  • Comparable Accuracy to DDS: Physics-constrained surrogates (PCS) achieve comparable relative L2 error and R² scores to data-driven surrogates (DDS) for predicting pressure fields, even when trained with significantly less data or no output data at all.
  • Faster Training with CNNs: CNN-based PCS models train significantly faster and achieve better accuracy in resolving fine details of flux fields compared to FC-NNs, especially for inputs with multiscale features. For instance, a CNN achieved better results than an FC-NN after 250 iterations, while the FC-NN required 2000 iterations.
  • Improved Generalization: PCS models show better generalization performance on out-of-distribution test inputs (e.g., warped GRF and channelized fields) compared to DDS models.
  • Accurate Uncertainty Quantification: The probabilistic surrogate, trained using the reverse KL divergence, successfully captures predictive uncertainty and demonstrates good calibration, as evidenced by reliability diagrams. The predictive mean for the pressure field achieved a relative L2 error of 0.0038, comparable to the deterministic surrogate.

Limitations and Future Work

  • Computational Cost of Probabilistic Models: While the probabilistic surrogate captures uncertainty, it can be at the expense of the accuracy of the mean prediction. The parameter $\beta$ controls this trade-off between accuracy and diversity.
  • Generalization to Unknowns: The paper notes that the current surrogate cannot effectively express what it "does not know," meaning it doesn't always show larger uncertainty when tested on inputs far from the training distribution.
  • Future Work: The authors suggest several promising directions:
  • Extending the framework to surrogate modeling for dynamical systems.
  • Improving generalization to out-of-distribution inputs through techniques like fine-tuning and learned gradient updates.
  • Combining physics-aware and data-driven approaches when limited simulation data and partial physics knowledge are available.
  • Scaling flow-based generative models to higher dimensions.
  • Developing more reliable probabilistic models that can better indicate when they are uncertain.
  • Exploring ways to enhance FC-NNs for capturing multiscale features.
  • Investigating the solution landscape with conditional generative surrogates.

Summary updated

Abstract

Highlights • Physics-constrained surrogates achieve comparable accuracy with data-driven ones without labeled data and generalize better. • Flow-based conditional generative model trained with reverse KL-divergence without labels captures the predictive uncertainty. • The developed models are used to solve PDEs, as surrogate models and for uncertainty propagation and calibration tasks. • Convolutional neural nets capture multiscale features of PDE solution fields much more effectively than fully-connected ones. Abstract Surrogate modeling and uncertainty quantification tasks for PDE systems are most often considered as supervised learning problems where input and output data pairs are used for training. The construction of such emulators is by definition a small data problem which poses challenges to deep learning approaches that have been developed to operate in the big data regime. Even in cases where such models have been shown to have good predictive capability in high dimensions, they fail to address constraints in the data implied by the PDE model. This paper provides a methodology that incorporates the governing equations of the physical model in the loss/likelihood functions. The resulting physics-constrained, deep learning models are trained without any labeled data (e.g. employing only input data) and provide comparable predictive responses with data-driven models while obeying the constraints of the problem at hand. This work employs a convolutional encoder-decoder neural network approach as well as a conditional flow-based generative model for the solution of PDEs, surrogate model construction, and uncertainty quantification tasks. The methodology is posed as a minimization problem of the reverse Kullback-Leibler (KL) divergence between the model predictive density and the reference conditional density, where the later is defined as the Boltzmann-Gibbs distribution at a given inverse temperature with the underlying potential relating to the PDE system of interest. The generalization capability of these models to out-of-distribution input is considered. Quantification and interpretation of the predictive uncertainty is provided for a number of problems.
BibTeX
@article{zhu2019physics,
  author = {Yinhao Zhu and Nicholas Zabaras and Phaedon-Stelios Koutsourelakis and Paris Perdikaris},
  title = {{Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data}},
  journal = {Journal of Computational Physics},
  year = {2019},
  doi = {10.1016/j.jcp.2019.05.024},
}
Figure