Bayesian time series classification

Peter Sykacek, Stephen J. RobertsNeurIPS 2001 · OpenReview

Introduction

In traditional signal processing, classification typically follows a rigid, two-step pipeline: first, features (such as the coefficients of an Autoregressive model) are extracted from raw data, and then these features are fed into a classifier (like a Hidden Markov Model).

However, this "broken link" approach is often suboptimal. The classifier treats the extracted features as absolute ground truth, ignoring any noise or uncertainty introduced during the extraction phase. Conversely, the feature extractor operates without knowledge of the classifier's objectives. To address this, the authors propose a unified Bayesian framework for time series classification that links these two stages probabilistically. By replacing fixed feature estimates with a "latent feature space," the model accounts for uncertainty and allows information to flow in both directions: raw data informs the features, while the classifier’s predictions act as a prior to refine feature extraction.

Methodology: A Unified Bayesian Framework

The authors develop a Gaussian and Multinomial observation Hidden Markov Model (GMOHMM) to jointly model feature extraction and classification.

Graphical representation of the proposed hierarchical Bayesian model.

The technical approach is broken down into several key components:

  1. The Latent Feature Space: Instead of relying on point estimates, features are treated as latent (hidden) variables. The authors use Autoregressive (AR) models to represent adjacent segments of the time series.
  2. Reparameterization: Because AR coefficients can be unstable, they are transformed into reflection coefficients using the Levinson algorithm, followed by an arctanh transformation. This maps the bounded coefficients to an infinite real line, making them suitable for modeling with Gaussian distributions.
  3. Model Order Selection: Recognizing that signal complexity varies, the model employs Reversible Jump Markov Chain Monte Carlo (RJMCMC). This allows the model to dynamically "jump" between different dimensions, learning the optimal AR model order for each segment on the fly.
  4. Inference: Given the highly non-linear nature of the joint likelihood, Gibbs sampling is used to estimate the posterior distributions of the hidden states, features, and model parameters.

Experimental Setup

The proposed model was evaluated on two primary types of data:

  • Synthetic Data: A first-order Markov sequence generating AR processes. To increase the difficulty, "artifacts"—segments of white noise completely uncorrelated with the class labels—were introduced.
  • EEG Data: Brainwave recordings from five subjects performing different cognitive tasks, including:
  • Auditory vs. Navigation: Mentally navigating a familiar environment based on sound.
  • Motor Imagination: Imagining moving the left or right hand. The EEG data was recorded from three electrode sites (T4, P4, C3, C4) at 384 Hz.

Key Insights

  • The Classifier as a Prior: A major advantage of this joint model is that the generative classifier acts as a prior for the feature extraction stage. For instance, if the HMM infers that the subject is likely in a "motor imagination" state, it biases the feature extractor toward coefficients typical of that state. This temporal context is crucial for resolving ambiguity in noisy signals.
  • Fusing Information by Certainty: The Bayesian approach naturally weights information based on its reliability. If a specific time series segment is highly noisy, the model's uncertainty increases, prompting it to rely more heavily on the Markov transition probabilities (the "history") to make a classification.
  • Handling Artifacts: Treating features as latent variables makes the model remarkably robust to noise. In synthetic tests, the model maintained high performance even when 20% of the data was replaced with meaningless white noise.

Results

The authors compared three experimental settings: Conditioning (using traditional point estimates), Marginalizing Features (integrating over feature values), and Full Integration (integrating over both features and model orders).

Classification accuracy across different experimental settings, comparing standard conditioning to full Bayesian integration.

In every scenario, the "Full Integration" approach—which treats both features and model complexity as random variables—resulted in a significant improvement in generalization accuracy compared to standard conditioning. For example, accuracy on synthetic data jumped from 75.3% to 91.5%, and similar gains were observed in the real-world EEG tasks.

Limitations and Future Directions

The primary trade-off of this approach is computational complexity. Utilizing MCMC techniques to integrate over latent variables is significantly more demanding than simple maximum likelihood estimation. Although the algorithm's complexity remains polynomial relative to the number of segments, it is not as lightweight as traditional methods.

However, the authors suggest that this "latent feature" philosophy extends beyond time series analysis. It could be generalized to almost any hierarchical model where feature extraction and classification are currently treated as separate silos. Future work may involve scaling this framework to multi-sensor settings, where spatial context could be integrated into the latent space alongside temporal dependencies.

Summary updated

Abstract

This paper proposes an approach to classification of adjacent segments of a time series as being either of classes. We use a hierarchical model that consists of a feature extraction stage and a generative classifier which is built on top of these features. Such two stage approaches are often used in signal and image processing. The novel part of our work is that we link these stages probabilistically by using a latent feature space. To use one joint model is a Bayesian requirement, which has the advantage to fuse information according to its certainty. The classifier is implemented as hidden Markov model with Gaussian and Multinomial observation distributions defined on a suitably chosen representation of autoregressive models. The Markov dependency is mo- tivated by the assumption that successive classifications will be corre- lated. Inference is done with Markov chain Monte Carlo (MCMC) tech- niques. We apply the proposed approach to synthetic data and to classi- fication of EEG that was recorded while the subjects performed different cognitive tasks. All experiments show that using a latent feature space results in a significant improvement in generalization accuracy. Hence we expect that this idea generalizes well to other hierarchical models.
BibTeX
@inproceedings{sykacek2001bayesian,
  author = {Peter Sykacek and Stephen J. Roberts},
  title = {{Bayesian time series classification}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2001},
  url = {https://openreview.net/forum?id=HJWZKLb_Wr},
}
Figure