Adaptive weighting of Bayesian physics informed neural networks for multitask and multiscale forward and inverse problems

Sarah Perez, Suryanarayana Maddu, Ivo F. Sbalzarini, Philippe Poncet

Abstract

In this paper, we present a novel methodology for automatic adaptive weighting of Bayesian Physics-Informed Neural Networks (BPINNs), and we demonstrate that this makes it possible to robustly address multi-objective and multi-scale problems. BPINNs are a popular framework for data assimilation, combining the constraints of Uncertainty Quantification (UQ) and Partial Differential Equation (PDE). The relative weights of the BPINN target distribution terms are directly related to the inherent uncertainty in the respective learning tasks. Yet, they are usually manually set a-priori, that can lead to pathological behavior, stability concerns, and to conflicts between tasks which are obstacles that have deterred the use of BPINNs for inverse problems with multi-scale dynamics. The present weighting strategy automatically tunes the weights by considering the multi-task nature of target posterior distribution. We show that this remedies the failure modes of BPINNs and provides efficient exploration of the optimal Pareto front. This leads to better convergence and stability of BPINN training while reducing sampling bias. The determined weights moreover carry information about task uncertainties, reflecting noise levels in the data and adequacy of the PDE model. We demonstrate this in numerical experiments in Sobolev training, and compare them to analytically ε-optimal baseline, and in a multi-scale Lokta-Volterra inverse problem. We eventually apply this framework to an inpainting task and an inverse problem, involving latent field recovery for incompressible flow in complex geometries.

Introduction

Physics-Informed Neural Networks (PINNs) have emerged as a cornerstone of scientific machine learning, embedding physical laws—typically expressed as partial differential equations (PDEs)—directly into the learning objective. Bayesian PINNs (BPINNs) extend this by providing uncertainty quantification (UQ). However, a significant bottleneck in BPINN deployment is the "multi-objective weighting" problem. Balancing data fidelity, PDE residuals, and boundary conditions often requires extensive manual hyperparameter tuning, especially in multiscale problems where gradients can differ by orders of magnitude. If one goal dominates the gradient energy, others are ignored, leading to biased results or complete training failure.

The AW-HMC Methodology

To address these challenges, we introduce Adaptively Weighted Hamiltonian Monte Carlo (AW-HMC). This method automates the balancing of task-specific weights by considering the multi-task nature of the target posterior distribution.

Gradient Variance Balancing

The core of AW-HMC lies in its ability to monitor the variances of the gradients for each objective task. Instead of static weights, the algorithm dynamically adjusts weights during the "burning" or warmup phase of the MCMC sampling. This ensures that no single task dominates the gradient energy, allowing for a more stable and balanced exploration of the parameter space.

Evolution of gradient variances for different tasks during AW-HMC training.

By normalizing the influence of each objective, AW-HMC effectively regularizes the target distribution. This prevents the "stiffness" issues common in multiscale problems, where standard samplers like the No-U-Turn Sampler (NUTS) often fail by excessively reducing step sizes to near-zero.

Experimental Results

The effectiveness of AW-HMC was demonstrated across several benchmarks, ranging from synthetic Sobolev training to complex fluid dynamics.

Sobolev Training and Uncertainty Quantification

In Sobolev training benchmarks involving high-order derivatives, AW-HMC successfully recovered weights that closely matched analytically ε-optimal values. The resulting Bayesian Model Averaging (BMA) provides not only accurate predictions but also robust uncertainty estimates, as shown in the 1D regression task below.

BMA prediction versus exact solution with uncertainty bounds.

Multiscale and Inverse Problems

The method was further validated on a Lotka-Volterra predator-prey system, where it successfully inferred parameters across three orders of magnitude—a task where standard HMC and NUTS failed. In fluid dynamics simulations of stenotic blood flow, the model performed latent field recovery. Using only partial velocity measurements, it accurately estimated the Reynolds number and recovered hidden pressure fields from sparse, noisy data with a 95% Prediction Interval Coverage Probability (PICP).

Discussion and Conclusion

AW-HMC provides a robust framework for BPINNs, turning task weights into indicators of inherent uncertainty. A lower weight for a specific task effectively signals higher noise or a less adequate model for that component. While the method relies on a stable burning phase and carries the computational overhead typical of MCMC, it eliminates the need for manual tuning and enables the solution of previously intractable multiscale inverse problems. Future work will explore its application to non-linear, time-dependent PDEs and heteroscedastic noise modeling.

Summary updated

BibTeX
@article{perez2023adaptive,
  author = {Sarah Perez and Suryanarayana Maddu and Ivo F. Sbalzarini and Philippe Poncet},
  title = {{Adaptive weighting of Bayesian physics informed neural networks for multitask and multiscale forward and inverse problems}},
  journal = {Journal of Computational Physics},
  year = {2023},
  doi = {10.1016/j.jcp.2023.112342},
}
Figure