Hidden Markov Mixture of Gaussian Process Functional Regression: Utilizing Multi-Scale Structure for Time-Series Forecasting

Tao Li, Jinwen MaICLR 2023 · OpenReview

TL;DR

This paper introduces the Hidden Markov Mixture of Gaussian Process Functional Regression (HM-GPFR) and its Bayesian extension (BHM-GPFR) to improve time series forecasting by explicitly modeling multi-scale temporal structures. These models outperform existing methods, especially in long-term forecasting and when dealing with "cold start" scenarios, by leveraging both fine-grained (Gaussian Process) and coarse-grained (Hidden Markov Model) temporal dependencies.

What Problem Does This Paper Address?

Traditional mixture models for functional data, like the mixture of Gaussian Process Functional Regressions (mix-GPFR), often assume that individual time series or sample curves are generated independently. However, in many real-world scenarios, such as electricity load forecasting, there's a temporal dependency across these "batches" of data. This means the underlying patterns or "states" of the time series can evolve over longer periods, and assuming independence misses this crucial coarse-grained temporal structure. This limitation is particularly problematic in "cold start" situations where initial data points for a new period are missing, making it difficult to infer the current underlying pattern.

Methods

The core innovation is the integration of a Hidden Markov Model (HMM) into a Gaussian Process Functional Regression (GPFR) mixture framework.

  • GPFR: At the fine level, Gaussian Processes are used to model the temporal dependencies within each individual time series or sample curve.
  • HMM: At the coarse level, an HMM is employed to capture the temporal dynamics and state transitions between these time series or sample curves. This allows the model to learn how the underlying patterns evolve over time.
  • HM-GPFR: This model combines GPFR for fine-level structure and HMM for coarse-level structure, treating the entire process as a state-switching dynamic.
  • BHM-GPFR: To enhance robustness and address potential overfitting, a Bayesian approach is introduced by placing prior distributions on model parameters, leading to the Bayesian Hidden Markov-based GPFR mixture model.
  • Learning: The Expectation-Maximization (EM) algorithm is adapted for parameter learning in both HM-GPFR and BHM-GPFR. For BHM-GPFR, a variational inference approach is used due to increased complexity.

Datasets

The primary dataset used for training and evaluation is an electricity load dataset from the State Grid of China, specifically for a city in northwest China. This dataset consists of 96 samples per day, recorded every 15 minutes, for a total of 365 daily records. The authors use the 2010 data for training and then perform rolling forecasts on subsequent data. No new benchmark datasets are introduced.

Key Insights and Ideas

  • Multi-Scale Temporal Structure is Crucial: The paper strongly demonstrates that explicitly modeling both fine-grained (within-day patterns) and coarse-grained (day-to-day evolution) temporal structures significantly improves time series forecasting accuracy, especially for longer horizons.
  • HMM Solves the "Cold Start" Problem: By incorporating an HMM, the model can infer the likely current state even with missing initial observations for a new period, overcoming a major limitation of previous mix-GPFR models.
  • Bayesian Approach Enhances Robustness: The BHM-GPFR, by using prior distributions, shows improved robustness and better performance, particularly in mitigating overfitting compared to the non-Bayesian HM-GPFR.
  • Interpretability of Evolving Patterns: The learned hidden states and transition probabilities provide meaningful insights into the underlying electricity consumption patterns and how they evolve over time, offering valuable interpretability for practical applications.

Key Results

  • Improved Accuracy: HM-GPFR and BHM-GPFR consistently outperformed classical time series methods (AR, MA, ARIMA, SARMA) and machine learning methods (LSTM, FNN, SVR, EGPM) in both short-term and long-term forecasting.
  • Long-Term Forecasting Advantage: The proposed models showed a significant advantage in long-term forecasting compared to mix-GPFR, mix-GPNM, and DPM-GPFR.
  • "Cold Start" Performance: In cold start scenarios (e.g., predicting the first day of a new period with no prior observations), HM-GPFR and BHM-GPFR achieved higher accuracy than mix-GPFR and its variants.
  • BHM-GPFR Superiority: BHM-GPFR generally performed better than HM-GPFR, indicating the benefits of the Bayesian approach in terms of robustness and prediction accuracy.
  • Interpretability: The models successfully identified distinct electricity load patterns (modes) and their transition dynamics, which were visualized and analyzed.

Limitations and Future Work

  • Hyper-parameter Sensitivity: While the paper explores the sensitivity to the number of components (K), it notes that K ≤ 10 generally yields stable results. However, determining the optimal K in a data-driven way remains an interesting direction.
  • Computational Cost: The EM algorithm and variational inference can be computationally intensive. The authors suggest reducing computational cost by introducing inducing points for GP regression.
  • Model Selection: While expert knowledge can guide the choice of K, a data-driven model selection strategy is a promising area for future research. The authors suggest hierarchical-Dirichlet-process-based hidden Markov models as a potential solution.
  • Generalizability: The paper focuses on electricity load forecasting. Further research could explore the applicability of these models to other types of multi-scale time series data.

Summary updated

Abstract

The mixture of Gaussian process functional regressions (GPFRs) assumes that there are a batch of time-series or sample curves which are generated by independent random processes with different temporal structures. However, in the real situations, these structures are actually transferred in a random manner from a long time scale. Therefore, the assumption of independent curves is not true in practice. In order to get rid of this limitation, we propose the hidden Markov based GPFR mixture model (HM-GPFR) by describing these curves with both fine and coarse level temporal structures. Specifically, the temporal structure is described by the Gaussian process model at the fine level and hidden Markov process at the coarse level. The whole model can be regarded as a random process with state switching dynamics. To further enhance the robustness of the model, we also give a priori to the model parameters and develop Bayesian hidden Markov based GPFR mixture model (BHM-GPFR). Experimental results demonstrate that the proposed methods have both high prediction accuracy and good interpretability.
BibTeX
@inproceedings{li2023hidden,
  author = {Tao Li and Jinwen Ma},
  title = {{Hidden Markov Mixture of Gaussian Process Functional Regression: Utilizing Multi-Scale Structure for Time-Series Forecasting}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=QP4nkeQ1BpT},
}
Figure