Mean-Field Variational Inference for Gradient Matching with Gaussian Processes

Nico S. Gorbach, Stefan Bauer, Joachim M. BuhmannarXiv 2016 · DOI · arXiv

Abstract

Gradient matching with Gaussian processes is a promising tool for learning parameters of ordinary differential equations (ODEs). The essence of gradient matching is to model the prior over state variables as a Gaussian process, which implies that the joint distribution given the ODEs and GP kernels is also Gaussian distributed. The state-derivatives are integrated out analytically since they are modeled as latent variables. However, the state variables themselves are also latent variables because they are contaminated by noise. Previous work sampled the state variables since integrating them out is not analytically tractable. In this paper, we use mean-field approximation to establish tight variational lower bounds that decouple state variables and are therefore, in contrast to the integral over state variables, analytically tractable and even concave for a restricted family of ODEs, including nonlinear and periodic ODEs. Such variational lower bounds facilitate "hill climbing" to determine the maximum a posteriori estimate of ODE parameters. An additional advantage of our approach over sampling methods is the determination of a proxy to the intractable posterior distribution over state variables given observations and the ODEs.

Introduction

Ordinary Differential Equations (ODEs) are fundamental to modeling dynamic systems in biology, physics, and engineering. However, a significant bottleneck in using these models is parameter estimation—identifying the specific constants (such as reaction rates) that allow the model to fit observed data. Traditionally, this requires numerically solving the ODEs repeatedly within an optimization loop, a process that is computationally prohibitive for complex or high-dimensional systems.

"Gradient matching" has emerged as a popular alternative. Instead of solving the ODE directly, this approach smooths the noisy observations—often using Gaussian Processes (GPs)—and attempts to match the derivative of the smoothed data to the derivative predicted by the ODE. While efficient, this method faces a major hurdle: the true states of the system are latent and noisy. Integrating out these latent states to find the marginal likelihood is analytically intractable, forcing researchers to rely on computationally expensive Markov Chain Monte Carlo (MCMC) sampling. This paper introduces a Mean-Field Variational Inference (MFVI) framework to transform this sampling problem into a stable and efficient optimization task.

Methodology

Gaussian Process Prior and Product of Experts

The proposed framework begins by modeling the state variables as a Gaussian Process. Because the derivative of a GP is itself a GP, the relationship between a state and its rate of change can be handled analytically. The authors employ a "Product of Experts" approach, combining the data-driven GP smoothing with the dynamics-driven ODE model. This creates a joint distribution that accounts for both the observed noise and the underlying physical laws.

Mean-Field Approximation

The core innovation of this work is the application of Mean-Field Variational Inference to the gradient matching problem. By establishing a variational lower bound (ELBO), the authors decouple the state variables across different time points. This decoupling is the key to tractability; it allows the previously "impossible" integral over the latent states to be solved analytically.

Optimization via EM-like Scheme

The authors develop an Expectation-Maximization (EM) style optimization scheme:

  1. E-step: Update the "proxy" distribution ($Q$) of the hidden states to better approximate the true posterior.
  2. M-step: Perform "hill climbing" (gradient ascent) to find the ODE parameters ($ heta$) that maximize the lower bound.

For a specific class of ODEs—those following mass-action kinetics where terms are products of states—the states appear quadratically in the mathematical formulation. In these cases, the proxy distribution is Gaussian, and the optimization objective is often concave, ensuring that the algorithm converges efficiently to a global or high-quality local optimum.

Experimental Evaluation

The method was evaluated on two benchmark systems to test its accuracy and robustness against noise.

Lotka-Volterra System

As a classic predator-prey model, the Lotka-Volterra system is characterized by non-linear, periodic oscillations. The MFVI approach successfully recovered the true parameters ($ heta_1$ through $ heta_4$) even when the observations were corrupted by significant Gaussian noise (variance levels of 0.1 and 0.25). This demonstrates the method's ability to handle non-linear dynamics that are traditionally difficult for gradient matching.

Protein Signaling Transduction Pathway

This more complex biological model involves five states and multiple reaction rates, including Michaelis-Menten kinetics. The authors demonstrated that such kinetics can be mathematically transformed to fit the mass-action requirement of their framework. While the complexity of the pathway made exact parameter recovery more challenging, the method successfully preserved the relative ranking of parameter values and provided a highly accurate proxy distribution that captured the true underlying dynamics of the protein states.

Discussion and Key Insights

Efficiency and Stability

By replacing MCMC sampling with a deterministic optimization (hill climbing), the MFVI approach avoids the high computational overhead and convergence monitoring issues typical of sampling-based gradient matching. The decoupling of states via mean-field approximation provides a clear path to scalability.

The State Proxy

Beyond parameter estimation, the method provides a posterior proxy ($Q$). This serves as a denoised estimate of the system's true trajectory, offering researchers insight into the hidden dynamics of the system that are not directly visible in the noisy raw data.

Limitations and Future Directions

While powerful, the current proof of tractability relies on the ODEs having a specific functional form (polynomial or mass-action kinetics). While many systems can be transformed into this form, others involving complex rational functions may require further approximation. Additionally, the current study focuses on fully observable systems; extending this framework to partially observable systems—where some state variables are never measured—remains a primary goal for future research.

Conclusion

This work presents a significant step forward in making ODE parameter estimation practical for complex systems. By leveraging Mean-Field Variational Inference, the authors provide a framework that is not only faster than traditional sampling methods but also provides theoretical guarantees of concavity for a wide range of biological and physical models.

Summary updated

Abstract

Gradient matching with Gaussian processes is a promising tool for learning parameters of ordinary differential equations (ODE's). The essence of gradient matching is to model the prior over state variables as a Gaussian process which implies that the joint distribution given the ODE's and GP kernels is also Gaussian distributed. The state-derivatives are integrated out analytically since they are modelled as latent variables. However, the state variables themselves are also latent variables because they are contaminated by noise. Previous work sampled the state variables since integrating them out is \textit{not} analytically tractable. In this paper we use mean-field approximation to establish tight variational lower bounds that decouple state variables and are therefore, in contrast to the integral over state variables, analytically tractable and even concave for a restricted family of ODE's, including nonlinear and periodic ODE's. Such variational lower bounds facilitate "hill climbing" to determine the maximum a posteriori estimate of ODE parameters. An additional advantage of our approach over sampling methods is the determination of a proxy to the intractable posterior distribution over state variables given observations and the ODE's.
BibTeX
@misc{gorbach2016mean,
  author = {Nico S. Gorbach and Stefan Bauer and Joachim M. Buhmann},
  title = {{Mean-Field Variational Inference for Gradient Matching with Gaussian Processes}},
  year = {2016},
  eprint = {1610.06949},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1610.06949},
}
Figure