LAMORE: A Stable, Scalable Approach to Latent Vector Autoregressive Modeling of Categorical Time Series

Yubin Park, Carlos Carvalho, Joydeep GhoshAISTATS 2014

Introduction

Categorical time series—sequences of discrete labels such as medical diagnoses or purchase histories—are ubiquitous in modern data science. Unlike continuous numerical data, categorical observations are inherently "lossy," providing limited information about the complex underlying states of a system. Latent Vector Autoregressive (VAR) models are frequently employed to capture these hidden dynamics. However, standard estimation techniques, typically involving the Expectation-Maximization (EM) algorithm paired with particle filters, face significant hurdles: instability due to observational noise and poor scalability as particle counts increase.

Methodology

The LAMORE framework addresses these challenges through three primary innovations designed to transform unstable, memory-intensive estimation into a robust and fast process:

  • Asymptotic Mean Regularization (AMOR): By penalizing deviations between the model's theoretical stationary mean and the observed sample mean, AMOR acts as a stabilizing force. This "gyroscope" effect prevents the optimization process from diverging due to local noise.
  • Pseudo-Bayesian Update (BAM): Utilizing a sequential update mechanism inspired by Bayesian linear regression, LAMORE processes particles individually. This removes the memory bottleneck associated with batch updates, making runtime nearly independent of particle volume.
  • Low-resolution Augmentation: Training on both original and coarse-grained versions of the time series helps mitigate the variance and bias inherent in Monte Carlo sampling.

Experimental Results

The authors evaluated LAMORE against synthetic and real-world datasets, including Medicare Part-D claims and the MIMIC-II ICU database.

Efficiency and Scalability

A critical advantage of LAMORE is its computational efficiency. While traditional brute-force EM (BPEM) exhibits a sharp, non-linear increase in runtime as the number of particles grows—often leading to memory exhaustion—LAMORE maintains a nearly constant, flat runtime regardless of particle count.

Figure 1: Comparison of elapsed time per iteration versus the number of particles for BPEM, AMORE, and LAMORE.

Stability and Convergence

On simulated data, LAMORE demonstrates superior stability. While baseline EM methods frequently fail to converge or diverge significantly from ground truth parameters, LAMORE consistently reaches the true values with minimal variance across various parameter types (e.g., constants and transition matrices).

Figure 2: Parameter convergence over iterations. LAMORE and AMORE variants track the 'True' parameter values more closely and stably than the baseline model.

In real-world applications, such as predicting pharmaceutical brand loyalty in Medicare data, LAMORE significantly outperformed standard "lagged" logistic regression models (glmnet). By capturing the underlying latent loyalty of beneficiaries, LAMORE provided actionable insights into system "inertia" that black-box models often miss.

Conclusion

LAMORE provides a robust and interpretable framework for categorical time series analysis. By leveraging asymptotic properties and sequential updates, it overcomes the traditional trade-offs between model complexity and computational feasibility. While currently limited by an assumption of stationarity, future work may integrate Markov switching models to address dynamically changing environments.

Summary updated

BibTeX
@inproceedings{park2014lamore,
  author = {Yubin Park and Carlos Carvalho and Joydeep Ghosh},
  title = {{LAMORE: A Stable, Scalable Approach to Latent Vector Autoregressive Modeling of Categorical Time Series}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2014},
}
Figure