TL;DR
This paper introduces the Hidden Markov Mixture of Gaussian Process Functional Regression (HM-GPFR) and its Bayesian extension (BHM-GPFR) to improve time series forecasting by explicitly modeling multi-scale temporal structures. These models outperform existing methods, especially in long-term forecasting and when dealing with "cold start" scenarios, by leveraging both fine-grained (Gaussian Process) and coarse-grained (Hidden Markov Model) temporal dependencies.
What Problem Does This Paper Address?
Traditional mixture models for functional data, like the mixture of Gaussian Process Functional Regressions (mix-GPFR), often assume that individual time series or sample curves are generated independently. However, in many real-world scenarios, such as electricity load forecasting, there's a temporal dependency across these "batches" of data. This means the underlying patterns or "states" of the time series can evolve over longer periods, and assuming independence misses this crucial coarse-grained temporal structure. This limitation is particularly problematic in "cold start" situations where initial data points for a new period are missing, making it difficult to infer the current underlying pattern.
Methods
The core innovation is the integration of a Hidden Markov Model (HMM) into a Gaussian Process Functional Regression (GPFR) mixture framework.
- GPFR: At the fine level, Gaussian Processes are used to model the temporal dependencies within each individual time series or sample curve.
- HMM: At the coarse level, an HMM is employed to capture the temporal dynamics and state transitions between these time series or sample curves. This allows the model to learn how the underlying patterns evolve over time.
- HM-GPFR: This model combines GPFR for fine-level structure and HMM for coarse-level structure, treating the entire process as a state-switching dynamic.
- BHM-GPFR: To enhance robustness and address potential overfitting, a Bayesian approach is introduced by placing prior distributions on model parameters, leading to the Bayesian Hidden Markov-based GPFR mixture model.
- Learning: The Expectation-Maximization (EM) algorithm is adapted for parameter learning in both HM-GPFR and BHM-GPFR. For BHM-GPFR, a variational inference approach is used due to increased complexity.
Datasets
The primary dataset used for training and evaluation is an electricity load dataset from the State Grid of China, specifically for a city in northwest China. This dataset consists of 96 samples per day, recorded every 15 minutes, for a total of 365 daily records. The authors use the 2010 data for training and then perform rolling forecasts on subsequent data. No new benchmark datasets are introduced.
Key Insights and Ideas
- Multi-Scale Temporal Structure is Crucial: The paper strongly demonstrates that explicitly modeling both fine-grained (within-day patterns) and coarse-grained (day-to-day evolution) temporal structures significantly improves time series forecasting accuracy, especially for longer horizons.
- HMM Solves the "Cold Start" Problem: By incorporating an HMM, the model can infer the likely current state even with missing initial observations for a new period, overcoming a major limitation of previous mix-GPFR models.
- Bayesian Approach Enhances Robustness: The BHM-GPFR, by using prior distributions, shows improved robustness and better performance, particularly in mitigating overfitting compared to the non-Bayesian HM-GPFR.
- Interpretability of Evolving Patterns: The learned hidden states and transition probabilities provide meaningful insights into the underlying electricity consumption patterns and how they evolve over time, offering valuable interpretability for practical applications.
Key Results
- Improved Accuracy: HM-GPFR and BHM-GPFR consistently outperformed classical time series methods (AR, MA, ARIMA, SARMA) and machine learning methods (LSTM, FNN, SVR, EGPM) in both short-term and long-term forecasting.
- Long-Term Forecasting Advantage: The proposed models showed a significant advantage in long-term forecasting compared to mix-GPFR, mix-GPNM, and DPM-GPFR.
- "Cold Start" Performance: In cold start scenarios (e.g., predicting the first day of a new period with no prior observations), HM-GPFR and BHM-GPFR achieved higher accuracy than mix-GPFR and its variants.
- BHM-GPFR Superiority: BHM-GPFR generally performed better than HM-GPFR, indicating the benefits of the Bayesian approach in terms of robustness and prediction accuracy.
- Interpretability: The models successfully identified distinct electricity load patterns (modes) and their transition dynamics, which were visualized and analyzed.
Limitations and Future Work
- Hyper-parameter Sensitivity: While the paper explores the sensitivity to the number of components (K), it notes that K ≤ 10 generally yields stable results. However, determining the optimal K in a data-driven way remains an interesting direction.
- Computational Cost: The EM algorithm and variational inference can be computationally intensive. The authors suggest reducing computational cost by introducing inducing points for GP regression.
- Model Selection: While expert knowledge can guide the choice of K, a data-driven model selection strategy is a promising area for future research. The authors suggest hierarchical-Dirichlet-process-based hidden Markov models as a potential solution.
- Generalizability: The paper focuses on electricity load forecasting. Further research could explore the applicability of these models to other types of multi-scale time series data.
Summary updated