Unified Inference for Variational Bayesian Linear Gaussian State-Space Models

David Barber, Silvia ChiappaNeurIPS 2006 · OpenReview

Introduction

Linear Gaussian State-Space Models (LGSSMs)—frequently referred to as Kalman Filters or Linear Dynamical Systems—are foundational tools in time-series analysis. They are widely used to track hidden states from noisy observations across diverse fields, from aerospace engineering to bioinformatics.

While standard LGSSMs assume fixed model parameters, adopting a Bayesian approach treats these parameters as uncertain distributions. This probabilistic treatment is highly advantageous: it prevents overfitting and allows the model to automatically "prune" unnecessary components. However, implementing the Variational Bayesian (VB) approximation for these models is notoriously difficult. The primary bottleneck is inferring the hidden state sequence, which introduces complex mathematical terms that do not fit the standard Kalman filter template. Historically, researchers have had to write custom, error-prone "Belief Propagation" algorithms from scratch, missing out on decades of numerical stability optimizations built into standard Kalman filter libraries.

This paper introduces a unified framework that simplifies VB inference in LGSSMs, allowing practitioners to use standard, off-the-shelf Kalman Filtering and Smoothing routines.

The "Mean + Fluctuation" Decomposition

The core challenge in VB inference for LGSSMs is that we no longer have a single transition matrix; instead, we have an expected value and its variance (fluctuations). This variance adds extra terms to the probability equations.

The authors solve this by introducing a clever mathematical remapping called the "Mean + Fluctuation Decomposition." By creating an augmented observation vector (padding the real data with zeros) and an augmented emission matrix (incorporating the Cholesky decomposition of the parameter covariances), the complex VB problem is transformed back into a standard LGSSM format.

Once transformed, any standard Kalman Filter or Smoother can be applied to solve the unified problem. This breakthrough allows practitioners to leverage established, numerically stable algorithms—such as square-root filters or Rauch-Tung-Striebel recursions—without reinventing the wheel. Furthermore, it demonstrates that Bayesian Independent Component Analysis (ICA) can be treated as a special, constrained case of this unified LGSSM framework.

Experimental Validation

The authors evaluated their unified framework on both synthetic and real-world datasets to demonstrate its efficacy and automatic model selection capabilities.

Synthetic Data and Automatic Pruning

In a controlled synthetic environment, three hidden sources were mixed into three observations. The goal was to test if the model could correctly identify the true number of underlying processes.

Because the Bayesian framework naturally performs "Automatic Relevance Determination," it can automatically prune unnecessary components. If initialized with a surplus of possible sources, the VB updates effectively "turn off" the redundant ones. The VB approach successfully identified the three original sources and pruned away the extras, whereas a standard Maximum A Posteriori (MAP) approach failed to estimate hyperparameters correctly and returned an incorrect number of sources.

Synthetic data results showing source separation and automatic pruning of unnecessary components.

Real-World EEG Source Separation

The method was also applied to real-world electroencephalogram (EEG) data recorded from four channels during a motor imagery task. EEG data is notoriously noisy, making signal separation a significant challenge.

The unified VB model successfully extracted independent dynamical processes from the noisy signals. It isolated low-frequency drifts and specific rhythmic activities (such as 10Hz and 20Hz bands) related to the mental tasks, while automatically ignoring the 50Hz electrical mains noise.

EEG data analysis isolating independent dynamical processes, including rhythmic activities and noise.

Limitations and Future Directions

While the unified framework offers significant practical advantages, it does come with trade-offs:

  • Computational Overhead: The augmentation process increases the dimensionality of the matrices. For models with extremely high-dimensional hidden states, this could lead to increased memory usage and computational cost.
  • Prior Sensitivity: The performance of the model, particularly in very "sparse" data regimes, can be sensitive to the choice of hyperparameters for the standard Gamma and Gaussian priors used.

Despite these limitations, the unified framework bridges a crucial gap between sophisticated Bayesian theory and practical engineering tools. The authors suggest that this approach could eventually be extended to even more complex models, such as switching state-space models or non-linear systems, paving the way for broader applications of Variational Bayesian methods.

Summary updated

Abstract

Linear Gaussian State-Space Models are widely used and a Bayesian treatment of parameters is therefore of considerable interest. The approximate Variational Bayesian method applied to these models is an attractive approach, used successfully in applications ranging from acoustics to bioinformatics. The most challenging aspect of implementing the method is in performing inference on the hidden state sequence of the model. We show how to convert the inference problem so that standard Kalman Filtering/Smoothing recursions from the literature may be applied. This is in contrast to previously published approaches based on Belief Propagation. Our framework both simplifies and unifies the inference problem, so that future applications may be more easily developed. We demonstrate the elegance of the approach on Bayesian temporal ICA, with an application to finding independent dynamical processes underlying noisy EEG signals. 1 Linear Gaussian State-Space Models Linear Gaussian State-Space Models (LGSSMs)1 are fundamental in time-series analysis [1, 2, 3]. In these models the observations v1:T 2 are generated from an underlying dynamical system on h1:T according to: v v vt = B ht + t , t N (0V , V ), h h ht = Aht-1 + t , t N (0H , H ) , where N (, ) denotes a Gaussian with mean and covariance , and 0X denotes an X dimensional zero vector. The observation vt has dimension V and the hidden state ht has dimension H . Probabilistically, the LGSSM is defined by: p(v1:T , h1:T |) = p(v1 |h1 )p(h1 ) tT p(vt |ht )p(ht |ht-1 ), =2 with p(vt |ht ) = N (B ht , V ), p(ht |ht-1 ) = N (Aht-1 , H ), p(h1 ) = N (, ) and where = {A, B , H , V , , } denotes the model parameters. Because of the widespread use of these models, a Bayesian treatment of parameters is of considerable interest [4, 5, 6, 7, 8]. An exact implementation of the Bayesian LGSSM is formally intractable [8], and recently a Variational Bayesian (VB) approximation has been studied [4, 5, 6, 7, 9]. The most challenging part of implementing the VB method is performing inference over h1:T , and previous authors have developed their own specialized routines, based on Belief Propagation, since standard LGSSM inference routines appear, at first sight, not to be applicable. 1 2 Also called Kalman Filters/Smoothers, Linear Dynamical Systems. v1:T denotes v1 , . . . , vT . A key contribution of this paper is to show how the Variational Bayesian treatment of the LGSSM can be implemented using standard LGSSM inference routines. Based on the insight we provide, any standard inference method may be applied, including those specifically addressed to improve numerical stability [2, 10, 11]. In this article, we decided to describe the predictor-corrector and Rauch-Tung-Striebel recursions [2], and also suggest a small modification that reduces computational cost. The Bayesian LGSSM is particularly of interest when strong prior constraints are needed to find adequate solutions. One such case is in EEG signal analysis, whereby we wish to extract sources that evolve independently through time. Since EEG is particularly noisy [12], a prior that encourages sources to have preferential dynamics is advantageous. This application is discussed in Section 4, and demonstrates the ease of applying our VB framework. 2 Bayesian Linear Gaussian State-Space Models In the Bayesian treatment of the LGSSM, instead of considering the model parameters as fixed, ^ ^ we define a prior distribution p(|), where is a set of hyperparameters. Then: ^ ^ p(v1:T |) = p(v1:T |)p(|) . (1) In a full Bayesian treatment we would define additional prior distributions over the hyperparameters ^ . Here we take instead the ML-II (`evidence') framework, in which the optimal set of hyperpa^ ^ rameters is found by maximizing p(v1:T |) with respect to [6, 7, 9]. For the parameter priors, here we define Gaussians on the columns of A and B 3 : p(A|, H ) jH e- j 2
BibTeX
@inproceedings{barber2006unified,
  author = {David Barber and Silvia Chiappa},
  title = {{Unified Inference for Variational Bayesian Linear Gaussian State-Space Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2006},
  url = {https://openreview.net/forum?id=Sk-BeFW_br},
}
Figure