Repulsive Ensembles for Bayesian Inference in Physics-informed Neural Networks

Philipp Pilar, Markus Heinonen, Niklas WahlströmarXiv 2025 · DOI · arXiv

Abstract

Physics-Informed Neural Networks (PINNs) are powerful tools for solving differential equations but often struggle with reliable uncertainty quantification, typically collapsing to a single solution. We introduce RE-PINN (Repulsive Ensembles for PINNs), a method that incorporates a mathematical repulsive force into the loss function. This force prevents ensemble members from clustering at a single local optimum, enabling them to accurately map the true Bayesian posterior distribution of both the solution and the underlying physical parameters.

Introduction

In inverse problems, where one seeks to discover both the system state and physical parameters from noisy data, a single point estimate is insufficient. Engineers and scientists require robust uncertainty estimates to assess the reliability of their models. While traditional Bayesian methods like Markov Chain Monte Carlo (MCMC) provide high-quality posteriors, they are computationally prohibitive for complex neural network models. Faster alternatives, such as standard Deep Ensembles or Variational Inference (VI), often fail in the context of PINNs. The physics-based constraints act as strong regularizers that pull ensemble members toward the same local optimum, leading to "posterior collapse"—a state where the model is overconfident in a potentially incorrect or incomplete answer.

Methodology: Repulsive Ensembles (RE-PINN)

RE-PINN treats ensemble members as particles in a potential field, minimizing the standard PINN loss while maintaining diversity through a repulsive term derived from Wasserstein gradient flow theory. This term effectively pushes ensemble members apart in regions where they are too crowded.

Joint Space Repulsion

A key contribution of this work is the necessity of repulsion in the joint space of function values ($f$) and differential equation parameters ($\lambda$). The authors demonstrate that repulsing only the function values is insufficient to prevent the collapse of inferred physical parameters.

Comparison of different repulsion strategies on the exponential equation. Only joint repulsion (f and $\lambda$) successfully recovers the true posterior distribution, matching the Monte Carlo baseline.

Density Approximation via KDE

To implement repulsion without knowing the true ensemble density, the authors utilize Kernel Density Estimation (KDE). To manage high-dimensional function spaces, "factorized" versions of this density are explored, treating different dimensions or parameters independently to maintain computational tractability and optimization stability.

Experimental Results

The researchers evaluated RE-PINN across several benchmarks, including exponential growth, damped harmonic oscillators, advection equations, and Lotka-Volterra dynamics.

Damped Harmonic Oscillator

In the damped harmonic oscillator problem, RE-PINN was tasked with inferring frequency ($\omega$) and damping ($\gamma$) from noisy observations. Unlike standard ensembles which collapsed to a single line, RE-PINN accurately captured the uncertainty in both the state and the physical parameters.

RE-PINN performance on the damped harmonic oscillator. The method accurately captures the uncertainty in both the time-series predictions and the physical parameters ($\gamma$ and $\omega$), closely matching the Monte Carlo ground truth.

Performance and Efficiency

Across all tests, RE-PINN uncertainty intervals closely matched MCMC "ground truth" baselines. Notably, the "fully factorized" variant achieved the highest log-likelihood on test data, indicating a superior probabilistic model. The computational overhead is minimal; for instance, training time on the exponential equation increased by only ~20% compared to non-repulsive ensembles.

Discussion and Insights

The study reveals that the physics loss in PINNs acts as a "diversity killer," making repulsion a requirement rather than an enhancement for Bayesian behavior. Furthermore, every member of a Repulsive Ensemble remains a valid PINN solution, ensuring that uncertainty intervals represent physically consistent states rather than arbitrary noise. Interestingly, the "fully factorized" model performed best in practice, likely due to the stability it provides during optimization compared to high-dimensional joint KDE.

Conclusion and Future Work

RE-PINN provides a scalable and effective framework for Bayesian inference in PINNs. Future research will explore using Gaussian Processes for density approximation to better capture function-space correlations and extending the model to infer measurement noise levels. Additionally, sequential training strategies could reduce the memory footprint required for large ensembles.

Reference: Pilar, P., Heinonen, M., & Wahlström, N. (2025). Repulsive Ensembles for Bayesian Inference in Physics-informed Neural Networks. arXiv:2505.17308v1.

Summary updated

BibTeX
@misc{pilar2025repulsive,
  author = {Philipp Pilar and Markus Heinonen and Niklas Wahlström},
  title = {{Repulsive Ensembles for Bayesian Inference in Physics-informed Neural Networks}},
  year = {2025},
  eprint = {2505.17308},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.17308},
}
Figure