Time Series Classification using the Hidden-Unit Logistic Model

Wenjie Pei, Hamdi Dibeklioglu, David M. J. Tax, Laurens van der MaatenarXiv 2015 · arXiv

Abstract

Time series classification is a fundamental task in computer vision and signal processing, yet traditional models like Hidden Conditional Random Fields (HCRFs) often struggle with a "parameter bottleneck." As the number of latent states increases to capture complex data structures, the parameter count grows linearly, leading to a high risk of overfitting. This paper presents the Hidden-Unit Logistic Model (HULM), a novel discriminative framework that replaces multinomial latent variables with a collection of binary stochastic hidden units. This approach allows for an exponential increase in latent state representation ($2^H$) with only a linear increase in parameters ($O(H)$), enabling more expressive models that remain computationally efficient and robust against overfitting.

Introduction

Assigning a single label to a sequence of observations—such as identifying a gesture from a video or a word from an audio clip—requires modeling complex temporal dependencies. While HCRFs have been widely used, their reliance on a single $k$-nomial variable at each time step limits their scalability. To increase the model's capacity, one must increase $k$, which directly increases the number of parameters. The Hidden-Unit Logistic Model (HULM) addresses this by using a distributed representation of latent states, decoupling the complexity of the state space from the growth of the parameter set.

The HULM Framework

The core innovation of HULM is its architecture, which utilizes $H$ binary stochastic hidden units at each time step $t$. Unlike traditional models that use a single state variable, HULM employs multiple parallel chains of binary units.

Graphical representation of the Hidden-Unit Logistic Model (HULM).

As shown in the figure above, the model consists of input observations $X_1, \dots, X_T$, hidden units $Z_{t,h}$, and a global label $y$. By using $H$ binary units, the model can represent $2^H$ distinct latent states. This distributed coding is significantly more efficient than the local representation used in HCRFs. For instance, representing 1024 states would require a 1024-way multinomial variable in an HCRF, but only 10 binary units in a HULM.

Inference and Learning

To ensure tractability, HULM is designed such that the hidden-unit chains are conditionally independent given the input sequence and the label. This independence allows for exact inference using a standard forward-backward algorithm, which scales linearly with both the sequence length $T$ and the number of hidden units $H$. Furthermore, HULM is a discriminative model, meaning it optimizes the conditional probability $P(y|X)$ directly, focusing its representational power on features that distinguish between classes rather than modeling the entire data distribution.

Experimental Evaluation

The HULM was evaluated on five diverse datasets spanning handwriting, speech, and action recognition, including Online Handwritten Characters (OHC), Arabic Spoken Digits (ASD), and the MSR Action 3D dataset. These datasets range from low-dimensional trajectory data to high-dimensional (570-dimensional) skeletal joint features.

Key Results

HULM consistently outperformed or matched state-of-the-art models like HCRF and Fisher Kernel Learning (FKL). Across all five datasets, it achieved the best average rank (1.8) and the lowest average generalization error (18.75%). Notably, on the Cohn-Kanade Extended (CK+) facial expression dataset, HULM reduced the error rate from 11.04% to 6.44%. It also demonstrated superior performance in detecting subtle facial Action Units, achieving state-of-the-art AUC and F1-scores.

Conclusion

The Hidden-Unit Logistic Model provides a powerful alternative to traditional latent-state models for time series classification. By leveraging binary stochastic units, it breaks the parameter bottleneck, allowing for highly expressive latent representations without the risk of overfitting associated with high-dimensional parameter spaces. Future work may involve extending the model to capture longer-range temporal dependencies through higher-order Markov chains or integrating it into multi-task learning frameworks for simultaneous sequence labeling and classification.

Summary updated

Abstract

We present a new model for time series classification, called the hidden-unit logistic model, that uses binary stochastic hidden units to model latent structure in the data. The hidden units are connected in a chain structure that models temporal dependencies in the data. Compared to the prior models for time series classification such as the hidden conditional random field, our model can model very complex decision boundaries because the number of latent states grows exponentially with the number of hidden units. We demonstrate the strong performance of our model in experiments on a variety of (computer vision) tasks, including handwritten character recognition, speech recognition, facial expression, and action recognition. We also present a state-of-the-art system for facial action unit detection based on the hidden-unit logistic model.
BibTeX
@misc{pei2015time,
  author = {Wenjie Pei and Hamdi Dibeklioglu and David M. J. Tax and Laurens van der Maaten},
  title = {{Time Series Classification using the Hidden-Unit Logistic Model}},
  year = {2015},
  eprint = {1506.05085},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1506.05085},
}
Figure