Assessment of uncertainty quantification in universal differential equations

Nina Schmid, David Fernandes del Pozo, Willem Waegeman, Jan Hasenauer

TL;DR

This paper tackles the crucial problem of quantifying uncertainty in Universal Differential Equations (UDEs), a powerful scientific machine learning framework that blends mechanistic models with neural networks. The authors formalize uncertainty quantification for UDEs and evaluate the performance of three key methods – ensembles, Variational Inference, and MCMC – on synthetic data, finding that ensembles and MCMC offer more reliable uncertainty estimates than Variational Inference.

What Problem Does This Paper Address?

Dynamical systems are often modeled using either mechanistic approaches (based on physical laws) or data-driven machine learning (ML) models. Scientific Machine Learning (SciML) aims to bridge this gap by integrating physical knowledge into ML models. Universal Differential Equations (UDEs) are a prime example, combining known mechanistic equations with neural networks to model the time derivatives of system states. This allows for incorporating prior knowledge and physical constraints, leading to more interpretable and generalizable models.

However, a critical aspect for the reliability and trustworthiness of these UDEs is uncertainty quantification (UQ). This involves understanding not only the model's predictions but also the uncertainty associated with its parameters and the predictions themselves. This uncertainty can arise from inherent randomness in the data (aleatoric uncertainty) or from limitations in our knowledge and model misspecifications (epistemic uncertainty). While UQ is well-studied in traditional ML and mechanistic modeling, its application to the unique challenges posed by UDEs – such as high-dimensional parameter spaces and complex loss landscapes – remains an open area. This paper aims to formalize UQ for UDEs and rigorously compare existing UQ methods in this context.

Methods

The authors propose a formal definition of uncertainty for UDEs, distinguishing between aleatoric and epistemic uncertainty. They then focus on epistemic uncertainty, which they decompose into model bias and approximation uncertainty. For this study, they assume the model form is fixed and focus on parameter uncertainty, which stems from not knowing the exact values of the mechanistic and neural network parameters.

They evaluate three primary UQ methods:

  • Ensemble-based UQ: This method draws inspiration from both deep learning ensembles and mechanistic modeling. It involves training multiple UDEs with different initializations and then selecting a subset of the best-performing models based on a likelihood-ratio test. This approach leverages the optimization process to explore the parameter space.
  • MCMC-based UQ: This approach treats the UDE as a Bayesian model and uses Markov Chain Monte Carlo (MCMC) sampling methods to approximate the posterior distribution of the parameters. They specifically investigate the No-U-Turn Sampler (NUTS) and parallel tempering algorithms, noting that parallel tempering showed better exploration of multimodal distributions.
  • Variational Inference (VI)-based UQ: This method approximates the posterior distribution of the parameters with a simpler, tractable distribution. They employ a mean-field approximation with multivariate normal base distributions.

Datasets

The paper does not introduce new datasets. Instead, it generates synthetic data from three different model formulations:

  • SEIR Pulse: A compartmental model for infectious diseases with a time-dependent transmission rate simulating a political intervention.
  • SEIR Waves: Another SEIR model with an oscillating transmission rate to represent complex time-dependent contact behavior.
  • Quadratic Dynamics: A simpler model used to explore scenarios where the neural network might dominate the dynamics.

These models are simulated using two different noise distributions: Gaussian and Negative Binomial, resulting in a total of eleven data scenarios. This synthetic approach allows for direct comparison of UQ methods against a known ground truth.

Key Insights and Ideas

  • Formalizing UQ for UDEs is crucial: The paper provides a much-needed formal framework for understanding and quantifying uncertainty in UDEs, which is essential for their reliable application.
  • Ensembles and MCMC outperform VI: The study reveals that ensemble-based and MCMC methods provide more reliable uncertainty estimates and predictions for UDEs compared to Variational Inference. This is particularly important given the potential for non-identifiabilities in UDEs.
  • Variational Inference struggles with UDEs: VI, while computationally efficient, produced biased estimates and underestimated uncertainty in the UDE context, even for small noise levels. This suggests that the assumptions underlying VI might not be well-suited for the complex posterior landscapes of UDEs.
  • Ensemble thresholds are tricky: While ensembles perform well, determining the appropriate threshold for selecting ensemble members remains a challenge and can significantly impact the resulting uncertainty bands.
  • MCMC is computationally expensive but interpretable: MCMC methods offer clear interpretability and good coverage of true parameters and dynamics, but at a significant computational cost.
  • Over-parameterization is a key challenge: The high dimensionality and potential symmetries in UDEs make parameter exploration difficult for MCMC and can lead to issues like overestimation of aleatoric uncertainty in ensembles.

Key Results

  • Ensemble and MCMC methods: Both ensemble and MCMC methods successfully contained the true parameters and simulated trajectories within their uncertainty and prediction intervals, respectively, across various noise levels.
  • Variational Inference results: VI consistently provided biased estimates for parameters and underestimated the uncertainty in trajectories, failing to cover true parameter values in some cases.
  • Computational efficiency: VI was the most computationally efficient, requiring less than 24 hours on a single CPU core. MCMC was significantly more demanding, taking 5-7 days for approximately 10,000 samples on a 20-core node. Ensembles also required substantial computation, with training 10,000 members taking 4-12 hours on 20 cores.
  • Parameter exploration: UMAP visualizations showed that ensembles clustered well, while MCMC (parallel tempering) exhibited a more patchy distribution, suggesting better exploration but potentially missing some high-quality optimized parameters.

Limitations and Future Work

  • Ensemble threshold selection: The paper acknowledges that defining a suitable threshold for ensemble selection requires further investigation.
  • MCMC convergence: MCMC methods, particularly in the context of UDEs, still face challenges with convergence and mixing, especially for multimodal posteriors.
  • Computational cost: The computational demands of MCMC and ensembles remain a significant hurdle for scaling to larger problems.
  • Model structure uncertainty: The current study focuses on parameter uncertainty. The authors suggest that investigating model structure uncertainty (i.e., how mechanistic terms are absorbed by the neural network) could provide valuable insights.
  • Hybrid UQ approaches: Exploring hybrid methods that combine the efficiency of ensembles with the interpretability of MCMC is proposed as a promising direction.
  • Automatic symmetry removal: Developing methods for automatic symmetry removal in UDEs could improve computational efficiency and enable scaling.
  • Broader applicability: The findings are expected to extend beyond UDEs to other SciML approaches, including ANNs for control and neural stochastic differential equations.

Summary updated

Abstract

Scientific machine learning is a new class of approaches that integrate physical knowledge and mechanistic models with data-driven techniques to uncover the governing equations of complex processes. Among the available approaches, universal differential equations (UDEs) combine prior knowledge in the form of mechanistic formulations with universal function approximators, such as neural networks. Integral to the efficacy of UDEs is the joint estimation of parameters for both the mechanistic formulations and the universal function approximators using empirical data. However, the robustness and applicability of these resultant models hinge upon the rigorous quantification of uncertainties associated with their parameters and predictive capabilities. In this work, we provide a formalization of uncertainty quantification (UQ) for UDEs and investigate key frequentist and Bayesian methods. By analyzing three synthetic examples of varying complexity, we evaluate the validity and efficiency of ensembles, variational inference and Markov-chain Monte Carlo sampling as epistemic UQ methods for UDEs.This article is part of the theme issue 'Uncertainty quantification for healthcare and biological systems (Part 2)'.
BibTeX
@inproceedings{schmid2025assessment,
  author = {Nina Schmid and David Fernandes del Pozo and Willem Waegeman and Jan Hasenauer},
  title = {{Assessment of uncertainty quantification in universal differential equations}},
  booktitle = {Philosophical},
  year = {2025},
  doi = {10.1098/rsta.2024.0444},
}
Figure