The neural moving average model for scalable variational inference of state space models

Thomas Ryder, Dennis Prangle, Andrew Golightly, Isaac MatthewsUAI 2021 · OpenReview

Abstract

Variational inference has had great success in scaling approximate Bayesian inference to big data by exploiting mini-batch training. To date, however, this strategy has been most applicable to models of independent data. We propose an extension to state space models of time series data based on a novel generative model for latent temporal states: the neural moving average model. This permits a subsequence to be sampled without drawing from the entire distribution, enabling training iterations to use mini-batches of the time series at low computational cost. We illustrate our method on autoregressive, Lotka-Volterra, FitzHugh-Nagumo and stochastic volatility models, achieving accurate parameter estimation in a short time.

Introduction

State Space Models (SSMs) are fundamental tools for sequential data analysis across diverse fields such as economics and neuroscience. These models assume a hidden state $x$ evolves over time, observed through noisy data $y$. However, standard inference methods like Kalman or particle filters require a full pass through the data ($O(T)$) to evaluate the likelihood, making them prohibitively slow for modern datasets where $T > 10^6$. While Variational Inference (VI) has enabled "big data" scaling for independent samples via mini-batching, it has struggled with time series because most flexible sequence models are autoregressive. In such models, sampling a state at time $t$ requires sampling every preceding state, breaking the ability to perform efficient local updates.

The Neural Moving Average (nMA) Model

The authors propose the Neural Moving Average (nMA) model to introduce a "locality property" to latent variables, enabling $O(1)$ mini-batch training.

The Locality Property

Instead of an autoregressive structure, the nMA uses a convolutional neural network (CNN) to process a sequence of independent noise variables. Because the CNN has a fixed receptive field (window size), any state $x_i$ only depends on a small window of noise variables around it. This allows the model to sample a subsequence from any point in a massive time series at a constant computational cost.

Normalizing Flows and Mini-batch VI

The nMA is structured as a type of Inverse Autoregressive Flow (IAF), transforming simple Gaussian noise into complex latent states through learned affine layers. By decomposing the log-density of the nMA model into local contributions, the authors derive an unbiased estimate of the Evidence Lower Bound (ELBO) gradient using only short subsequences. To enhance flexibility, the model incorporates both "global" side information (parameters $\theta$) and "local" side information (observed data $y$) into the CNN at every time step.

Experimental Evaluation

The nMA model was evaluated on several benchmarks:

  • Simulated AR(1) Model: Testing scaling up to $T=100,000$.
  • Lotka-Volterra (Predator-Prey): A non-linear SDE model with complex dynamics.
  • FitzHugh-Nagumo: A model of neuron membrane potentials with unobserved components.
  • Real-world Financial Data: Weekly U.S. Treasury bill rates (1967–1996) for Stochastic Volatility modeling.

Key Results

Constant Scaling and Efficiency

A primary advantage of the nMA model is its scalability. For the AR(1) model, the number of iterations required for convergence remained constant even as the total sequence length $T$ increased significantly. In terms of wall-clock time, the nMA achieved accurate parameter estimation on the Lotka-Volterra model in approximately 10–20 minutes, compared to nearly 2 hours for previous autoregressive VI methods.

Convergence analysis across varying sequence lengths (T) and receptive fields (l).

Accuracy and Latent State Inference

The marginal posteriors produced by nMA were found to be virtually indistinguishable from those produced by "gold standard" MCMC methods. Furthermore, the model successfully handled sparse data regimes, learning to accurately "impute" missing states between observations. This is particularly evident in complex models like Lotka-Volterra, where the model effectively captures latent trajectories.

Latent state trajectories and marginal posterior densities for the Lotka-Volterra model.

Conclusion

The Neural Moving Average model provides a scalable framework for Bayesian inference in SSMs by leveraging the locality property of CNNs. While it may struggle with systems exhibiting extremely long-range dependencies beyond its receptive field, its ability to perform $O(1)$ mini-batch updates makes it a powerful tool for massive time series datasets. Future work may explore multi-scale architectures to capture longer-range effects and more advanced flow transformations.

Summary updated

Abstract

Variational inference has had great success in scaling approximate Bayesian inference to big data by exploiting mini-batch training. To date, however, this strategy has been most applicable to models of independent data. We propose an extension to state space models of time series data based on a novel generative model for latent temporal states: the neural moving average model. This permits a subsequence to be sampled without drawing from the entire distribution, enabling training iterations to use mini-batches of the time series at low computational cost. We illustrate our method on autoregressive, Lotka-Volterra, FitzHugh-Nagumo and stochastic volatility models, achieving accurate parameter estimation in a short time.
BibTeX
@inproceedings{ryder2021neural,
  author = {Thomas Ryder and Dennis Prangle and Andrew Golightly and Isaac Matthews},
  title = {{The neural moving average model for scalable variational inference of state space models}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=Z0TPd2LYMV},
}
Figure