Deep adaptive sampling for surrogate modeling without labeled data

Xili Wang, Kejun Tang, Jiayu Zhai, Xiaoliang Wan, Chao Yang

Abstract

Surrogate modeling is of great practical significance for parametric differential equation systems. In contrast to classical numerical methods, using physics-informed deep learning methods to construct simulators for such systems is a promising direction due to its potential to handle high dimensionality, which requires minimizing a loss over a training set of random samples. However, the random samples introduce statistical errors, which may become the dominant errors for the approximation of low-regularity and high-dimensional problems. In this work, we present a deep adaptive sampling method for surrogate modeling ($DAS^2$), where we generalize the deep adaptive sampling (DAS) method to build surrogate models for low-regularity parametric differential equations. In the parametric setting, the residual loss function can be regarded as an unnormalized probability density function (PDF) of the spatial and parametric variables. This PDF is approximated by a deep generative model, from which new samples are generated and added to the training set. Since the new samples match the residual-induced distribution, the refined training set can further reduce the statistical error in the current approximate solution. We demonstrate the effectiveness of $DAS^2$ with a series of numerical experiments, including the parametric lid-driven 2D cavity flow problem with a continuous range of Reynolds numbers from 100 to 1000.

Introduction

In engineering and physics, solving differential equations across a range of parameters is a fundamental task—whether simulating airflow over a wing at varying angles or modeling heat distribution in varying geometries. While traditional numerical solvers provide high accuracy, they are computationally expensive, requiring a complete re-simulation for every new parameter set.

Physics-Informed Neural Networks (PINNs) have emerged as a powerful "surrogate" alternative, learning to predict system states directly. However, PINNs often struggle with "low-regularity" problems—scenarios characterized by sharp gradients or complex small-scale structures. When trained using standard uniform or random collocation points, these models often miss critical physical features, leading to high statistical errors. This challenge is amplified in high-dimensional parameter spaces where the "curse of dimensionality" makes uniform sampling computationally infeasible.

The $DAS^2$ Framework

This work introduces $DAS^2$ (Deep Adaptive Sampling for Surrogate modeling), a novel framework designed to build fast, accurate simulators without requiring any pre-computed "labeled" data. The core innovation of $DAS^2$ is the treatment of the training process as an iterative loop between two distinct neural networks:

  1. The Surrogate Model: A neural network that takes both spatial coordinates ($x$) and parameters ($\xi$) as inputs to predict the system's state. It is trained by minimizing the PDE residual—the degree to which the prediction violates the underlying physical laws.
  2. The Generative Model (KRnet): Instead of selecting training points at random, $DAS^2$ treats the squared PDE residual as an unnormalized probability density function (PDF). It utilizes a KRnet—an invertible normalizing flow model—to approximate this PDF.

By drawing new training samples from the KRnet, the algorithm automatically places more "sensors" in regions where the physics is hardest to learn (i.e., where the residual is highest). The authors propose two specific strategies: Joint PDF sampling, which samples space and parameters simultaneously, and Marginal PDF sampling, which adaptively focuses on the parameter space when spatial complexity is relatively low.

Experimental Validation

As a "label-free" method, $DAS^2$ was evaluated on four classic physical benchmarks to test its robustness across different dimensions and complexities:

  • 1D Parametric ODE: A foundational exponential growth model used to verify the basic adaptive logic.
  • High-Dimensional Operator Learning: A dynamical system featuring an 8-dimensional parameter space, testing the model's ability to overcome the curse of dimensionality.
  • Parametric Optimal Control: A geometric optimization problem inspired by hyperthermia cancer treatment, where the domain shape itself serves as a parameter.
  • 2D Lid-Driven Cavity Flow: A standard fluid dynamics benchmark involving the Navier-Stokes equations with Reynolds numbers ($Re$) ranging from 100 to 1000.

Key Insights

Residual as a Roadmap

The research reinforces the principle that the PDE residual is an excellent proxy for error. By identifying where the residual is high, the model identifies exactly where more data is needed to resolve complex physics.

Generative Models vs. Heuristics

Unlike traditional methods like Residual-based Adaptive Refinement (RAR), which select the "worst" points from a pre-defined, fixed pool, the KRnet can generate an infinite variety of new points across high-dimensional space. This allows $DAS^2$ to explore the parameter space much more effectively than heuristic-based sampling.

Variance Reduction

By matching the sample distribution to the residual, $DAS^2$ effectively reduces the variance of the loss function estimator. This leads to a more stable optimization process and a "flatter" loss landscape, which is generally associated with better generalization in deep learning.

Results and Performance

The empirical results demonstrate significant advantages over standard PINN approaches:

  • Superior Accuracy: In the 8D operator learning task, $DAS^2$ achieved accuracy one order of magnitude higher than uniform sampling or standard RAR.
  • Handling High Physics Complexity: For the 2D Cavity flow at $Re=1000$, $DAS^2$ successfully captured complex vortices that uniform sampling failed to resolve.
  • Massive Computational Speedup: Once trained, the $DAS^2$ surrogate predicted fluid flow in just 0.02 seconds. Compared to the 309.94 seconds required by a traditional high-fidelity solver (FEniCS), this represents a speedup of over 15,000x.
  • Zero Labeled Data: Critically, these results were achieved without ever running a traditional solver to generate "ground truth" training data, making the method highly efficient for discovery in data-poor environments.

Limitations and Future Work

While $DAS^2$ shows great promise, the authors identify several areas for further development. The convergence rate of the combined surrogate and generative model optimization remains an open theoretical question. Additionally, while the KRnet is more efficient than generating labeled data, it does introduce computational overhead compared to simple random sampling. Future research will also look toward extending this adaptive logic to problems with extremely complex or moving boundary conditions, where the interior residual is not the only source of error.

Summary updated

Abstract

Surrogate modeling is of great practical significance for parametric differential equation systems. In contrast to classical numerical methods, using physics-informed deep learning methods to construct simulators for such systems is a promising direction due to its potential to handle high dimensionality, which requires minimizing a loss over a training set of random samples. However, the random samples introduce statistical errors, which may become the dominant errors for the approximation of low-regularity and high-dimensional problems. In this work, we present a deep adaptive sampling method for surrogate modeling ($\text{DAS}^2$), where we generalize the deep adaptive sampling (DAS) method [62] [Tang, Wan and Yang, 2023] to build surrogate models for low-regularity parametric differential equations. In the parametric setting, the residual loss function can be regarded as an unnormalized probability density function (PDF) of the spatial and parametric variables. This PDF is approximated by a deep generative model, from which new samples are generated and added to the training set. Since the new samples match the residual-induced distribution, the refined training set can further reduce the statistical error in the current approximate solution. We demonstrate the effectiveness of $\text{DAS}^2$ with a series of numerical experiments, including the parametric lid-driven 2D cavity flow problem with a continuous range of Reynolds numbers from 100 to 1000.
BibTeX
@article{wang2024deep,
  author = {Xili Wang and Kejun Tang and Jiayu Zhai and Xiaoliang Wan and Chao Yang},
  title = {{Deep adaptive sampling for surrogate modeling without labeled data}},
  journal = {J. Sci. Comput.},
  year = {2024},
  url = {https://openreview.net/forum?id=nzIWW3HZQ3},
}
Figure