Information Theory Inspired Pattern Analysis for Time-series Data

Yushan Huang, Yuchen Zhao, Alexander Capstick, Francesca Palermo, Hamed Haddadi, Payam M. BarnaghiarXiv 2023 · OpenReview

Abstract

Current methods for pattern analysis in time series mainly rely on statistical features or probabilistic learning and inference methods to identify patterns and trends in the data. Such methods do not generalize well when applied to multivariate, multi-source, state-varying, and noisy time-series data. To address these issues, we propose a highly generalizable method that uses information theory-based features to identify and learn from patterns in multivariate time-series data. To demonstrate the proposed approach, we analyze pattern changes in human activity data. For applications with stochastic state transitions, features are developed based on Shannon's entropy of Markov chains, entropy rates of Markov chains, entropy production of Markov chains, and von Neumann entropy of Markov chains. For applications where state modeling is not applicable, we utilize five entropy variants, including approximate entropy, increment entropy, dispersion entropy, phase entropy, and slope entropy. The results show the proposed information theory-based features improve the recall rate, F1 score, and accuracy on average by up to 23.01% compared with the baseline models and a simpler model structure, with an average reduction of 18.75 times in the number of model parameters.

Introduction

Internet of Things (IoT) devices, particularly in healthcare, generate massive volumes of time-series data. However, this data is notoriously difficult to process: it is often multivariate, multi-source, rapidly state-varying, and noisy. Current approaches generally fall into two categories: statistical/probabilistic methods, which often fail to generalize because they ignore essential temporal relationships, and Deep Neural Networks (DNNs). While powerful, DNNs are often "black boxes," making it difficult to interpret clinical predictions. Furthermore, they require significant computational resources that may not be available in edge-computing scenarios.

This research proposes a "middle way": a feature engineering pipeline that captures complex temporal patterns using information theory while remaining interpretable and computationally efficient. By replacing raw data inputs with entropy-based features, we can utilize simpler, more transparent model architectures.

Methodology

The core innovation lies in an Entropy-based Feature Engineering Pipeline. Entropy measures the uncertainty, complexity, or regularity of a signal. The approach is categorized based on the nature of the data:

Stochastic State Transitions (Markovian Data)

When data can be modeled as transitions between discrete states—such as a patient moving from the kitchen to the lounge—the system utilizes Shannon’s entropy, entropy rates, and von Neumann entropy. A Neural Estimator for Entropy Production (NEEP) is employed to estimate entropy in non-equilibrium systems. This allows for the quantification of correlations across both space (different rooms) and time (different days) simultaneously.

Markovian state transitions between rooms and the corresponding transition matrix used for entropy calculation.

Continuous Signals (Non-Markovian Data)

For continuous signals where state modeling is less applicable, such as EEG or ECG data, the pipeline employs five entropy variants:

  • Approximate Entropy (ApEn): Quantifies regularity and unpredictability.
  • Increment Entropy: Captures the magnitude and direction of changes between points.
  • Dispersion, Phase, and Slope Entropy: Quantify the "disorder" or complexity of the waveform.

Experimental Evaluation

The researchers validated their approach across three distinct domains: the Minder Dataset (remote dementia monitoring), the Epileptic Seizure Recognition Dataset (ESRD), and the PTB Diagnostic ECG Database (PTBDB).

Comparison of Seizure EEG and Normal EEG waveforms, illustrating the complex temporal patterns analyzed using entropy variants.

The results across all three datasets were consistently superior to baseline models:

  • Minder Dataset: The entropy-based LSTM improved the Recall rate by 23.01% and the F1 score by 20.04% compared to the baseline.
  • ESRD (Seizure) Dataset: The entropy-driven MLP reduced the number of model parameters by 5.86 times compared to a standard CNN, while achieving a higher accuracy of 98.10%.
  • PTBDB (Heart) Dataset: The model structure was reduced by a factor of 61.35 in terms of parameters compared to the baseline CNN, while improving accuracy to 98.66%.

Conclusion

By transforming raw, noisy sensor data into high-quality entropy features, this method allows simple Multi-Layer Perceptrons (MLP) or shallow LSTMs to outperform deep, complex models. This shift prioritizes interpretability—where a spike in entropy provides a clear signal to clinicians—without sacrificing performance. Future work will focus on automating feature selection and exploring these information-theoretic features in other IoT domains such as data security and intelligent systems.

Summary updated

Abstract

Current methods for pattern analysis in time series mainly rely on statistical features or probabilistic learning and inference methods to identify patterns and trends in the data. Such methods do not generalize well when applied to multivariate, multi-source, state-varying, and noisy time-series data. To address these issues, we propose a highly generalizable method that uses information theory-based features to identify and learn from patterns in multivariate time-series data. To demonstrate the proposed approach, we analyze pattern changes in human activity data. For applications with stochastic state transitions, features are developed based on Shannon's entropy of Markov chains, entropy rates of Markov chains, entropy production of Markov chains, and von Neumann entropy of Markov chains. For applications where state modeling is not applicable, we utilize five entropy variants, including approximate entropy, increment entropy, dispersion entropy, phase entropy, and slope entropy. The results show the proposed information theory-based features improve the recall rate, F1 score, and accuracy on average by up to 23.01% compared with the baseline models and a simpler model structure, with an average reduction of 18.75 times in the number of model parameters.
BibTeX
@misc{huang2023information,
  author = {Yushan Huang and Yuchen Zhao and Alexander Capstick and Francesca Palermo and Hamed Haddadi and Payam M. Barnaghi},
  title = {{Information Theory Inspired Pattern Analysis for Time-series Data}},
  year = {2023},
  eprint = {2302.11654},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.11654},
}
Figure