A Probabilistic State Space Model for Joint Inference from Differential Equations and Data

Jonathan Schmidt, Nicholas Krämer, Philipp Hennig

Introduction

In scientific computing, mechanistic models—typically expressed as Ordinary Differential Equations (ODEs)—are essential for describing the evolution of systems ranging from viral spread to planetary motion. However, these models often contain unknown parameters or "latent forces" that must be estimated from noisy, real-world data.

Traditionally, this inference is handled as a "black-box" optimization or sampling problem. Researchers guess parameters, solve the ODE numerically, compare the results to data, and iterate. This process, often utilizing Markov Chain Monte Carlo (MCMC), is computationally expensive because the ODE must be solved thousands of times. This paper introduces a novel method to solve the ODE and infer parameters simultaneously in a single pass, drastically reducing computation time from hours to under a minute.

Methodology: The ODE as a Data Source

The authors leverage the field of Probabilistic Numerics to frame ODE solving as a Bayesian filtering problem. Rather than treating the solver as a deterministic black box, they integrate the physics directly into a probabilistic state-space model.

  1. Augmented State-Space: The model defines a joint Gauss-Markov prior for both the system state $x(t)$ (the ODE solution) and the latent force $u(t)$ (the unknown parameters).
  2. Dual Likelihoods: The model is conditioned on two information sources: standard noisy observational data and "artificial" ODE constraints that force the state's derivative to match the ODE vector field.
  3. Extended Kalman Filter (EKF) and Smoothing: Because ODEs are frequently non-linear, an EKF linearizes the physics on the fly. This allows the model to update its belief about both parameters and states simultaneously. A subsequent backward pass (Rauch-Tung-Striebel smoother) refines the entire trajectory.
The augmented state-space model treats the ODE constraints and observational data as distinct likelihoods within a unified filtering framework.

This approach achieves $O(N)$ linear complexity relative to the number of time steps, making it viable for long time series and complex systems where traditional methods fail to scale.

Experimental Evaluation

The researchers evaluated their model on several simulated environments—the Van-der-Pol oscillator, Lotka-Volterra predator-prey dynamics, and a simulated SIRD epidemic model—before applying it to real-world COVID-19 data from Germany (Jan 2020 – May 2021).

Key Results

  • Efficiency: In COVID-19 experiments, the proposed method completed inference in under one minute, compared to hours for MCMC.
  • Non-Parametric Discovery: By modeling parameters like contact rates as time-varying processes rather than fixed constants, the model can "discover" external influences.
  • Real-World Alignment: When applied to German COVID-19 data, the inferred contact rate ($\beta$) aligned closely with major government policy shifts, such as the "lockdown light" in late 2020.
Application to COVID-19 data: The model accurately tracks infection counts (left) while simultaneously inferring a time-varying contact rate (right) that reflects real-world policy interventions.

Conclusion and Future Work

By treating physics as a source of information rather than a computational hurdle, this method enables efficient, one-pass inference for complex dynamical systems. While the current implementation relies on first-order Taylor expansions (EKF) which may struggle with highly chaotic systems, it represents a significant step toward merging mechanistic modeling with modern Bayesian inference. Future work may explore higher-order filters or more robust handling of physical constraints, such as ensuring state positivity in biological models.

Summary updated

Abstract

Mechanistic models with differential equations are a key component of scientific applications of machine learning. Inference in such models is usually computationally demanding because it involves repeatedly solving the differential equation. The main problem here is that the numerical solver is hard to combine with standard inference techniques. Recent work in probabilistic numerics has developed a new class of solvers for ordinary differential equations (ODEs) that phrase the solution process directly in terms of Bayesian filtering. We here show that this allows such methods to be combined very directly, with conceptual and numerical ease, with latent force models in the ODE itself. It then becomes possible to perform approximate Bayesian inference on the latent force as well as the ODE solution in a single, linear complexity pass of an extended Kalman filter / smoother — that is, at the cost of computing a single ODE solution. We demonstrate the expressiveness and performance of the algorithm by training, among others, a non-parametric SIRD model on data from the COVID-19 outbreak.
BibTeX
@inproceedings{schmidt2021probabilistic,
  author = {Jonathan Schmidt and Nicholas Krämer and Philipp Hennig},
  title = {{A Probabilistic State Space Model for Joint Inference from Differential Equations and Data}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=7e4FLufwij},
}
Figure