Assessment of Uncertainty Quantification in Universal Differential Equations

Nina Schmid, David Fernandes del Pozo, Willem Waegeman, Jan Hasenauer

Abstract

Scientific Machine Learning is a new class of approaches that integrate physical knowledge and mechanistic models with data-driven techniques for uncovering governing equations of complex processes. Among the available approaches, Universal Differential Equations (UDEs) are used to combine prior knowledge in the form of mechanistic formulations with universal function approximators, like neural networks. Integral to the efficacy of UDEs is the joint estimation of parameters within mechanistic formulations and the universal function approximators using empirical data. The robustness and applicability of resultant models, however, hinge upon the rigorous quantification of uncertainties associated with these parameters, as well as the predictive capabilities of the overall model or its constituent components. With this work, we provide a formalisation of uncertainty quantification (UQ) for UDEs and investigate important frequentist and Bayesian methods. By analysing three synthetic examples of varying complexity, we evaluate the validity and efficiency of ensembles, variational inference and Markov chain Monte Carlo sampling as epistemic UQ methods for UDEs.

Introduction

In the burgeoning field of Scientific Machine Learning (SciML), Universal Differential Equations (UDEs) have emerged as a powerful tool. They allow researchers to retain mechanistic equations based on first principles while utilizing neural networks to learn unknown terms in the dynamics. However, UDEs are often "over-parameterized," meaning multiple parameter combinations can explain the same data. For critical decision-making—such as predicting disease spread or mechanical failure—rigorous Uncertainty Quantification (UQ) is essential. While UQ is well-studied for pure neural networks and mechanistic models, this work fills the gap by formalizing UQ specifically for the hybrid nature of UDEs.

Methodology

The study investigates three primary epistemic UQ methods to quantify uncertainty stemming from data limitations:

  1. Ensemble-based UQ: A frequentist approach where the model is trained repeatedly (e.g., 10,000 times) from different random initializations. A likelihood-ratio test filters for high-quality runs that fit the data well.
  2. Markov chain Monte Carlo (MCMC): A fully Bayesian approach using the No-U-Turn Sampler (NUTS) and Parallel Tempering to map the entire probability distribution of parameters.
  3. Variational Inference (VI): An approximation method that fits a simple distribution (e.g., Gaussian) to the complex true parameter distribution, offering speed at the potential cost of accuracy.
Conceptual comparison of uncertainty quantification methods for Universal Differential Equations.

The framework distinguishes between aleatoric uncertainty (inherent data noise) and epistemic uncertainty (parameter uncertainty).

Experimental Evaluation

Methods were evaluated using three synthetic dynamical systems of increasing complexity:

  • Quadratic Dynamics: A simple system with two non-identifiable parameters.
  • SEIR Pulse: An epidemiological model (Susceptible-Exposed-Infectious-Recovered) where the transmission rate changes abruptly.
  • SEIR Waves: A complex version where the transmission rate follows a periodic, wave-like pattern.

These systems were tested across 10 data scenarios using Gaussian noise for continuous data and Negative Binomial noise for discrete count data.

Results and Discussion

The analysis revealed that neural networks naturally introduce symmetries and multiple "modes" (different parameter sets yielding similar results) into the loss landscape, a complexity inherited by UDEs.

  • MCMC Performance: Parallel tempering MCMC provided the most accurate and narrow uncertainty bands for mechanistic parameters. It successfully captured multiple modes, which proved crucial for generalizing to new initial conditions not seen during training. However, it was computationally intensive, requiring 5–7 days on 20 CPU cores.
  • Ensemble Performance: Ensembles were a strong, parallelizable alternative (4–12 hours on 20 cores), though they rely on somewhat arbitrary significance thresholds for model inclusion.
  • VI Limitations: While the fastest method (<24h on a single core), VI often failed to capture hidden dynamics in complex cases because its simplified distribution could not account for multi-modal parameter spaces.
Predictive uncertainty for the SEIR model across Ensemble, MCMC, and Variational Inference methods.

The choice of noise model (Gaussian vs. Negative Binomial) significantly impacted uncertainty band width, underscoring that UQ quality depends heavily on underlying statistical assumptions.

Conclusion

This work demonstrates that while MCMC is the gold standard for robust UQ in UDEs due to its ability to handle multi-modal landscapes, its computational cost remains a barrier. Ensembles offer a practical trade-off, while VI requires caution in complex scenarios. Future work should address model form uncertainty—the risk of fundamental errors in the mechanistic equations or neural network architecture—and explore hybrid UQ methods that combine the speed of ensembles with the precision of MCMC.

Summary updated

Abstract

Scientific Machine Learning is a new class of approaches that integrate physical knowledge and mechanistic models with data-driven techniques for uncovering governing equations of complex processes. Among the available approaches, Universal Differential Equations (UDEs) are used to combine prior knowledge in the form of mechanistic formulations with universal function approximators, like neural networks. Integral to the efficacy of UDEs is the joint estimation of parameters within mechanistic formulations and the universal function approximators using empirical data. The robustness and applicability of resultant models, however, hinge upon the rigorous quantification of uncertainties associated with these parameters, as well as the predictive capabilities of the overall model or its constituent components. With this work, we provide a formalisation of uncertainty quantification (UQ) for UDEs and investigate important frequentist and Bayesian methods. By analysing three synthetic examples of varying complexity, we evaluate the validity and efficiency of ensembles, variational inference and Markov chain Monte Carlo sampling as epistemic UQ methods for UDEs.
BibTeX
@inproceedings{schmid2025assessment,
  author = {Nina Schmid and David Fernandes del Pozo and Willem Waegeman and Jan Hasenauer},
  title = {{Assessment of Uncertainty Quantification in Universal Differential Equations}},
  booktitle = {Philosophical},
  year = {2025},
  doi = {10.1098/rsta.2024.0444},
}
Figure