A Fourier State Space Model for Bayesian ODE Filters

Hans Kersting, Maren MahsereciarXiv 2020 · OpenReview

TL;DR

This paper introduces a novel "Fourier state space model" for Bayesian ODE filters, aiming to improve the prediction of periodic solutions. They propose a hybrid approach combining this Fourier model with traditional Taylor-based models to achieve efficient global extrapolation for oscillating ODEs.

What Problem Does This Paper Address?

Solving ordinary differential equations (ODEs) is a fundamental task in many scientific and engineering fields. Bayesian ODE filtering offers a probabilistic approach to this, providing uncertainty quantification for the ODE solution. The standard method relies on Taylor expansions to predict the solution forward in time. However, many important ODEs, like those describing oscillators, have periodic solutions. Taylor expansions can struggle with long-term extrapolation of periodic functions, potentially requiring many small steps for accuracy. This paper questions whether Fourier expansions, which are naturally suited for periodic functions, can be integrated into Bayesian ODE filtering to improve long-term prediction efficiency.

Methods

The core contribution is the development of a Fourier state space model (SSM) for ODEs. This model represents the ODE solution using a set of harmonic oscillators, whose coefficients are inferred. This differs from the standard Taylor SSM, which uses an integrated Brownian motion prior and relies on Taylor expansions of the mean to extrapolate.

The authors then propose a hybrid Taylor-Fourier model. This approach splits the time domain into two parts:

  1. Learning Phase: On the initial interval $[0, T_p]$, the ODE is solved using the standard Taylor SSM. Simultaneously, the Fourier SSM is trained "on the fly" using the data generated by the Taylor model.
  2. Prediction Phase: On the subsequent interval $[T_p, T]$, the prediction is made solely using the trained Fourier SSM.

This hybrid strategy aims to leverage the local accuracy of Taylor expansions for learning the Fourier coefficients and then use the Fourier model for efficient global extrapolation of periodic solutions.

Datasets

Not discussed in the paper. The authors use synthetic ODEs for evaluation.

Key Insights and Ideas

  • Fourier expansions are a natural fit for periodic ODEs: Unlike Taylor expansions, which approximate locally, Fourier series provide a global representation for periodic functions. This suggests they could be more efficient for long-term predictions of oscillating systems.
  • The hybrid approach offers a path to efficient extrapolation: By first learning Fourier coefficients using a reliable Taylor-based method and then switching to the Fourier model for prediction, the authors hypothesize a way to achieve fast, accurate extrapolation without repeated evaluations of the ODE's vector field.
  • The "dynamic model" is key to ODE solvers: The paper reinforces the idea that the underlying dynamic model (the prior over the ODE solution) is fundamental to how ODE solvers work, determining their extrapolation behavior.

Key Results

The paper presents experimental results on two oscillating ODEs: the Van der Pol oscillator and the FitzHugh-Nagumo model. The plots in Figures 1 and 2 demonstrate that the hybrid ODE filter can indeed pick up structure from the trajectories and extrapolate using a sum of harmonic oscillators. However, the authors acknowledge that the quality of the extrapolation is "not good enough yet" and attribute this to an "ad-hoc choice of parameters" and potentially needing a larger number of Fourier modes ($J$).

Limitations and Future Work

The authors explicitly acknowledge several limitations and suggest future research directions:

  • Parameter Tuning: The current parameter choices (e.g., angular velocity $\omega_0$, number of Fourier modes $J$) are ad-hoc and not fine-tuned. Future work should focus on principled ways to select these parameters.
  • Extrapolation Quality: The extrapolation quality needs improvement. This might involve choosing $\omega_0$ more accurately to match the oscillator's period and increasing $J$.
  • Active Learning for Fourier Coefficients: The paper suggests exploring active learning strategies to determine which data points from the Taylor model are most informative for training the Fourier model.
  • Bayesian Quadrature: Given that Fourier coefficients are defined by integrals, the authors propose borrowing ideas from Bayesian quadrature to improve their estimation.
  • Broader Applicability: The authors hope this approach could be useful for long-time horizon predictions with limited computational budgets, potentially benefiting areas like reinforcement learning where real-time decision-making is crucial.

Summary updated

Abstract

Gaussian ODE filtering is a probabilistic numerical method to solve ordinary differential equations (ODEs). It computes a Bayesian posterior over the solution from evaluations of the vector field defining the ODE. Its most popular version, which employs an integrated Brownian motion prior, uses Taylor expansions of the mean to extrapolate forward and has the same convergence rates as classical numerical methods. As the solution of many important ODEs are periodic functions (oscillators), we raise the question whether Fourier expansions can also be brought to bear within the framework of Gaussian ODE filtering. To this end, we construct a Fourier state space model for ODEs and a `hybrid' model that combines a Taylor (Brownian motion) and Fourier state space model. We show by experiments how the hybrid model might become useful in cheaply predicting until the end of the time domain.
BibTeX
@misc{kersting2020fourier,
  author = {Hans Kersting and Maren Mahsereci},
  title = {{A Fourier State Space Model for Bayesian ODE Filters}},
  year = {2020},
  eprint = {2007.09118},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2007.09118},
}
Figure