Fast variational inference for nonparametric clustering of structured time-series

James Hensman, Magnus Rattray, Neil D. Lawrence

Abstract: In this publication, we combine two Bayesian non-parametric models: the Gaussian Process (GP) and the Dirichlet Process (DP). Our innovation in the GP model is to introduce a variation on the GP prior which enables us to model structured time-series data, i.e. data containing groups where we wish to model inter- and intra-group variability. Our innovation in the DP model is an implementation of a new fast collapsed variational inference procedure which enables us to optimize our variational approximation significantly faster than standard VB approaches. In a biological time series application we show how our model better captures salient features of the data, leading to better consistency with existing biological classifications, while the associated inference algorithm provides a twofold speed-up over EM-based variational inference.

Introduction

Clustering time-series data, such as gene expression levels, is a fundamental task in bioinformatics. However, researchers often face two significant challenges: accounting for complex hierarchical structures (like biological replicates) and ensuring the scalability of Bayesian nonparametric models. This paper introduces a "Structured GP-DP" model that addresses these issues by combining Gaussian Processes (GPs) with Dirichlet Processes (DPs) and employing a high-performance inference engine.

Methods: Structured GP-DP and Fast Inference

The authors propose a hierarchical model where a Dirichlet Process handles clustering, while the "atoms" of the DP are hierarchical Gaussian Processes. In this framework, each cluster possesses a "mean" behavior modeled by a GP, and individual replicates within that cluster are modeled as deviations from that mean. This explicit modeling of inter- and intra-group variability prevents the "over-clustering" common in simpler models that treat replicate differences as independent noise.

A Gaussian Process fit to a single cluster of time-series data, capturing the underlying trend and uncertainty.

To solve this model efficiently, the authors introduce Collapsed Variational Inference (CVB). By analytically marginalizing the GP functions and DP weights, they focus optimization solely on cluster assignments. Furthermore, they utilize Riemannian natural gradients instead of standard Variational Bayesian Expectation-Maximization (VBEM). This approach navigates the parameter space more effectively, allowing the optimizer to escape the flat plateaus that often stall traditional algorithms.

Experimental Evaluation

The model was evaluated on three distinct datasets:

  1. Synthetic Data: 10 clusters of sine waves with randomized frequencies and phases to test recovery of ground-truth structures.
  2. Drosophila Development: Gene expression data from six different species of fruit flies. This dataset features a natural hierarchy of species and replicates.
  3. Mouse Cartilage: 304 genes related to circadian rhythms, measured in duplicate over a 48-hour cycle, utilizing a periodic covariance function to incorporate prior knowledge.

Key Results and Insights

The hierarchical approach proved significantly more parsimonious and accurate than non-hierarchical alternatives. On the Drosophila dataset, the structured model identified 52 clusters, whereas a model treating replicates as noise produced 245 clusters—many of which were "phantom" clusters explaining correlated noise.

A selection of clusters identified by the Structured GP-DP model, demonstrating the variety of temporal patterns captured across the dataset.

The Riemannian optimization provided a substantial speedup, reaching convergence in nearly half the time required by VBEM on the mouse cartilage data (14s vs 34s). This efficiency, combined with the model's ability to automatically determine the number of clusters through a merge-split procedure, makes it a robust tool for large-scale biological data analysis.

Conclusion

By treating structure as a feature rather than noise and leveraging the geometry of the posterior distribution for optimization, the authors have developed a powerful framework for time-series clustering. The resulting model captures salient biological features more effectively than standard methods while providing a significant computational speed-up. Future work may extend this framework to other types of structured data, such as motion capture data.

Summary updated

Abstract

In this publication, we combine two Bayesian non-parametric models: the Gaussian Process (GP) and the Dirichlet Process (DP). Our innovation in the GP model is to introduce a variation on the GP prior which enables us to model structured time-series data, i.e. data containing groups where we wish to model inter- and intra-group variability. Our innovation in the DP model is an implementation of a new fast collapsed variational inference procedure which enables us to optimize our variationala pproximation significantly faster than standard VB approaches. In a biological time series application we show how our model better captures salient features of the data, leading to better consistency with existing biological classifications, while the associated inference algorithm provides a twofold speed-up over EM-based variational inference.
BibTeX
@article{hensman2015fast,
  author = {James Hensman and Magnus Rattray and Neil D. Lawrence},
  title = {{Fast variational inference for nonparametric clustering of structured time-series}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2015},
  url = {https://openreview.net/forum?id=HX4TOBeupr},
}
Figure