Predicting multiple future steps in a time series—such as electricity demand for the next 24 hours—presents a dual challenge: capturing complex nonlinear relationships between inputs and outputs, and modeling the inherent dependencies between the output time steps themselves. Traditional linear models like ARIMA often fail to capture nonlinearities, while deep learning models like Recurrent Neural Networks (RNNs) frequently treat output steps as independent, neglecting the structured nature of the sequence.
To bridge this gap, researchers presented a hybrid architecture called CoR (Conditional Random Fields on top of Recurrent Neural Networks) at AAAI-18. This model integrates Sparse Gaussian Conditional Random Fields (SGCRF) with deep RNNs to create a system that is both a powerful feature extractor and a sophisticated reasoner about output dependencies.
The Challenge of Multi-Step Forecasting
Multi-step forecasting is difficult because of two primary factors:
- Nonlinear Input Correlations: The relationship between observed features (e.g., weather, historical usage) and the target is often highly nonlinear and temporally dependent.
- Structured Output Dependencies: Predicted values are not independent; for instance, energy demand at 10:00 AM is strongly correlated with demand at 11:00 AM.
While RNNs excel at the first challenge, they often struggle to explicitly model the "structure" of the entire output sequence. Conversely, SGCRFs are excellent at modeling output structure but typically require extensive manual feature engineering to handle raw, nonlinear input data.
The CoR Architecture
The CoR architecture consists of two primary modules stacked vertically to leverage the strengths of both deep learning and probabilistic graphical models.
- The Feature Extractor (Bottom): Stacked Bi-directional RNNs (or LSTMs) process the input sequence $X$ to produce an intermediate representation $z$. This automated feature extraction replaces the need for manual engineering, allowing the model to learn complex temporal patterns directly from raw data.
- The Structured Head (Top): An SGCRF takes the representation $z$ and predicts the final sequence $y$. It utilizes a precision matrix ($\Lambda$) to explicitly model the conditional dependencies between different time steps in the output sequence.
Joint Optimization and Training
Training this hybrid model is complex because the precision matrix in a Gaussian CRF must remain positive-definite. The authors found that end-to-end training using Projected Gradient Descent (PGD) was the most effective approach. During backpropagation, if the precision matrix becomes invalid, it is "projected" back onto the space of positive-definite matrices using eigenvalue decomposition. This joint optimization allows the RNN to learn representations specifically tailored for the structured reasoning of the CRF.
Experimental Evaluation and Results
The model was rigorously tested on synthetic datasets and the real-world NPower Forecasting Challenge 2016 dataset. In the NPower Challenge, the best CoR variant (using LSTMs and end-to-end training) achieved a Mean Absolute Percentage Error (MAPE) of 3.87%, significantly outperforming the competition's first-place benchmark of 7.13%.
Beyond accuracy, CoR offers substantial computational advantages. Pure SGCRFs struggle with high-dimensional features because their calculations scale quadratically. By using the RNN to compress inputs into a smaller intermediate state, CoR maintains efficiency even with large-scale data.
As shown in the table above, for long sequences ($T=200$), CoR is not only more accurate than a pure SGCRF but also significantly faster and more memory-efficient. Specifically, it demonstrated a 5x speedup and used approximately 80% less memory compared to the standalone SGCRF.
Conclusion
The CoR framework demonstrates that combining deep recurrent networks with structured graphical models provides a robust solution for complex time-series forecasting. By automating feature engineering through RNNs and enforcing output consistency through SGCRFs, the model achieves state-of-the-art performance with improved scalability. Future work may explore adapting this "RNN + Structured Head" approach to other domains like natural language processing or computer vision, where structured output is equally critical.
Summary updated