CoCoGen: Physically Consistent and Conditioned Score-Based Generative Models for Forward and Inverse Problems

Christian Jacobsen, Yilin Zhuang, Karthik DuraisamySIAM J. Sci. Comput 2023 · OpenReview

TL;DR

This paper introduces CoCoGen, a novel framework for training score-based generative models that enforce physical laws (like PDEs) during sample generation. This allows for more accurate and physically plausible simulations in forward and inverse problems, outperforming traditional methods in certain scenarios.

What Problem Does This Paper Address?

Generative models have shown great promise in various fields, but applying them to physics-based problems has been challenging. A key issue is ensuring that generated samples adhere to the underlying physical laws, such as partial differential equations (PDEs). Standard generative models, even those trained on physics data, can produce outputs that violate these laws, leading to non-physical results. This paper tackles the problem of how to efficiently enforce physical consistency in score-based generative models for both predicting system behavior (forward problems) and inferring system parameters from observations (inverse problems).

Methods

The core innovation lies in a physically-consistent sampling strategy. Instead of modifying the training process, the authors inject the governing PDE into the sampling phase. This is achieved by:

  • Unconditional Score-Based Models: They first train standard score-based generative models (using variance-preserving SDEs and probability flow ODEs) on physics data.
  • Physical Consistency Steps: During sampling, after the standard reverse diffusion process, they iteratively apply "correction steps." These steps involve calculating the PDE residual (how much the current sample violates the PDE) and then nudging the sample in the direction that minimizes this residual. This is done using the gradient of the residual with respect to the physical variables.
  • Conditional Augmentation (ControlNet-like): For specific tasks, they leverage a ControlNet-like architecture. This involves freezing a pre-trained unconditional model and training a smaller "conditional augmentation" module. This allows for efficient adaptation to tasks like field prediction or reconstruction using various conditioning inputs (e.g., generative parameters, sparse measurements) without retraining the entire large model.
  • RePaint (for Inverse Problems): For field reconstruction from sparse measurements, they also explore the RePaint technique, which alternates between forward and backward diffusion steps to better incorporate known (measured) information into the unknown regions.

Datasets

The paper primarily uses datasets generated from Darcy's law for fluid flow in porous media. They create datasets with varying intrinsic dimensionalities (s=16 and s=256) for the permeability field. They also use a 1D Burgers equation dataset for temporal PDE problems. These datasets are generated by solving the respective PDEs with specific parameters and boundary conditions. No new benchmarks are introduced; rather, existing physics problems are used to demonstrate the method's efficacy.

Key Insights and Ideas

  • Enforcing Physics at Inference Time: The most significant insight is that physical consistency can be effectively enforced after training, during the sampling process. This offers great flexibility, as the same trained model can be used for various PDEs by simply providing the appropriate residual calculation.
  • PDE Residual Minimization: Directly minimizing the PDE residual during sampling is a powerful way to ensure physical plausibility, even if the model's accuracy in predicting exact values is not perfect.
  • PF ODE vs. Reverse SDE: The Probability Flow Ordinary Differential Equation (PF ODE) solver generally leads to smaller physical residuals and requires fewer physical consistency steps compared to the reverse SDE, making it a preferred choice for fidelity-critical applications.
  • Denoising Effect: Applying physical consistency steps to the pressure field in Darcy flow also had a denoising effect on the permeability field, suggesting a synergistic benefit.
  • Conditional Models as Flexible Add-ons: The ControlNet-like approach for conditional generation is highly effective, allowing for rapid adaptation to new tasks by training only a small augmentation module on top of a frozen, powerful unconditional model.

Key Results

  • Reduced Physical Residuals: CoCoGen significantly reduces physical residuals, achieving values comparable to or even lower than conventional PDE solvers, especially when using the PF ODE. For Darcy flow, they achieved residuals as low as 3.0 (L1 norm) with just a few physical consistency steps.
  • Accurate Field Generation: In forward problems (e.g., Darcy flow field generation from parameters), the model generates samples that are visually and quantitatively accurate, closely matching ground truth.
  • Effective Field Reconstruction/Inversion: For inverse problems (e.g., inferring permeability from pressure measurements), the method shows strong performance, outperforming a POD-based method in terms of physical residual, especially with sparse measurements.
  • 1D Burgers Equation: The model accurately captures shock wave dynamics and achieves low residuals, demonstrating its applicability to hyperbolic PDEs.
  • Minimal Impact on Reconstruction Accuracy: While physical consistency steps reduce residuals, they have a minimal negative impact on the reconstruction accuracy itself, especially when measurements are available.

Limitations and Future Work

  • Hyperparameter Tuning: The effectiveness of physical consistency sampling depends on hyperparameters like the number of time steps (τ), physical consistency steps (N, M), and the step size (ε). Finding optimal values can still require some tuning.
  • Computational Cost: While more efficient than retraining, the physical consistency steps do add computational overhead to the sampling process. The RePaint method, while effective for inverse problems, significantly increases sampling cost.
  • Future Work: The authors suggest exploring more advanced SDE/ODE solvers for potentially further residual reduction and efficiency gains. They also believe their framework can be applied to a wider range of scientific machine learning tasks. They highlight the potential for using their method in scenarios where uncertainty quantification is crucial, especially in inverse problems with limited data.

Summary updated

Abstract

Recent advances in generative artificial intelligence have had a significant impact on diverse domains spanning computer vision, natural language processing, and drug discovery. This work extends the reach of generative models into physical problem domains, particularly addressing the efficient enforcement of physical laws and conditioning for forward and inverse problems involving partial differential equations (PDEs). Our work introduces two key contributions: firstly, we present an efficient approach to promote consistency with the underlying PDE. By incorporating discretized information into score-based generative models, our method generates samples closely aligned with the true data distribution, showcasing residuals comparable to data generated through conventional PDE solvers, significantly enhancing fidelity. Secondly, we showcase the potential and versatility of score-based generative models in various physics tasks, specifically highlighting surrogate modeling as well as probabilistic field reconstruction and inversion from sparse measurements. A robust foundation is laid by designing unconditional score-based generative models that utilize reversible probability flow ordinary differential equations. Leveraging conditional models that require minimal training, we illustrate their flexibility when combined with a frozen unconditional model. These conditional models generate PDE solutions by incorporating parameters, macroscopic quantities, or partial field measurements as guidance. The results illustrate the inherent flexibility of score-based generative models and explore the synergy between unconditional score-based generative models and the present physically-consistent sampling approach, emphasizing the power and flexibility in solving for and inverting physical fields governed by differential equations, and in other scientific machine learning tasks.
BibTeX
@article{jacobsen2023cocogen,
  author = {Christian Jacobsen and Yilin Zhuang and Karthik Duraisamy},
  title = {{CoCoGen: Physically Consistent and Conditioned Score-Based Generative Models for Forward and Inverse Problems}},
  journal = {SIAM J. Sci. Comput.},
  year = {2023},
  url = {https://openreview.net/forum?id=VcQJ0AkLpQ},
}
Figure