A Novel Gaussian Sum Smoother for Approximate Inference in Switching Linear Dynamical Systems

David Barber, Bertrand MesotNeurIPS 2006

Introduction

This paper introduces Expectation Correction (EC), a novel algorithm for approximate smoothed inference in Switching Linear Dynamical Systems (SLDS). By avoiding a common but restrictive approximation used in the classic Kim smoother, EC better integrates future information and provides a stable, single-pass alternative to more complex methods like Expectation Propagation (EP).

The Challenge of Switching Linear Dynamical Systems

Many real-world signals—such as human speech or financial markets—do not follow a single linear pattern. Instead, they "switch" between different modes. A Switching Linear Dynamical System (SLDS) models this by using a set of linear equations, where a discrete "switch" variable determines which equation to use at any given time.

Graphical model of an Augmented Switching Linear Dynamical System

The primary challenge is inference: if we only observe the noisy output, how do we deduce both the hidden continuous state and which "switch" was active? Because every time step could involve any switch state, the number of possible histories grows exponentially ($S^T$), making exact calculation impossible for even short sequences. Existing solutions like the "Kim smoother" make heavy approximations that lose accuracy, while "Expectation Propagation" (EP) can be numerically unstable and prone to crashing. Furthermore, in an Augmented SLDS (aSLDS), the switch itself can depend on the previous continuous state, adding to the complexity.

Expectation Correction (EC): A Two-Step Approach

To address these inference challenges, the authors propose a two-step approach called Expectation Correction (EC):

  1. Forward Pass (Filtering): The method uses standard Assumed Density Filtering (ADF). At each step, the distribution of the hidden state is represented as a mixture of Gaussians. To keep the computation manageable, this mixture is "collapsed" back down to a fixed number of components ($I$) after each step.
  2. Backward Pass (Smoothing): This is the core innovation. Traditional smoothers (like Kim's) assume that the current switch state is independent of future continuous states given the next switch state. EC discards this assumption. Instead, it uses numerical integration (or sampling) to more accurately "correct" the filtered estimates using information from the entire future sequence.

Unlike EP, which requires multiple iterations and complex parameter conversions, EC requires only a single forward and backward pass. It operates entirely using "moments" (means and covariances), making it significantly more stable.

Experimental Setup

The authors evaluated EC on two primary fronts:

  • Synthetic "Toy" Data: A controlled environment with a binary switch and 3D hidden states to compare EC against Particle Filters, Rao-Blackwellized Particle Filters, and EP.
  • TI-DIGITS Database: A standard speech recognition dataset containing recordings of digits (0-9 and "oh"). They used this to test the algorithm's performance in Noise Robust Automatic Speech Recognition (ASR) by artificially adding Gaussian noise to the recordings.

Key Results and Insights

The experiments revealed several critical advantages of the EC approach:

  • Overcoming the "Kim" Weakness: The classic Kim smoother's biggest flaw is an independence assumption that ignores how future continuous states might inform our guess of past switch states. EC proves that maintaining this coupling leads to significantly better state estimation. In toy experiments, EC consistently outperformed Kim smoothing and Particle Filters in correctly identifying hidden switch states.
  • Superior Stability: While Expectation Propagation (EP) is theoretically powerful, the authors found it frequently failed to converge or became numerically unstable. EC provides a "plug-and-play" alternative that is as stable as the standard Kalman Filter, remaining robust even for time-series spanning several thousand steps.
  • Speech in the Noise: On the TI-DIGITS task, the EC-based model (AR-SLDS) demonstrated incredible resilience to noise.
ASR accuracy comparison across different noise levels

As shown in the results, at a Signal-to-Noise Ratio (SNR) of nearly 0 dB (where the noise is as loud as the speech), standard HMMs dropped to a mere 9.1% accuracy. In stark contrast, the AR-SLDS with Expectation Correction maintained 61.2% accuracy.

Limitations and Future Work

While EC is highly robust, its accuracy still depends on the "forgetting" property of Markovian models. If a system has extremely long-range correlations, a small mixture of Gaussians might not be sufficient to capture the posterior distribution accurately.

Future work could involve exploring more sophisticated ways to "collapse" the Gaussian mixtures or applying the EC framework to even more complex "dynamic hybrid networks" where discrete and continuous variables interact in non-linear ways.

Summary updated

BibTeX
@inproceedings{barber2006novel,
  author = {David Barber and Bertrand Mesot},
  title = {{A Novel Gaussian Sum Smoother for Approximate Inference in Switching Linear Dynamical Systems}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2006},
}
Figure