Sparse Gaussian Conditional Random Fields on Top of Recurrent Neural Networks

Xishun Wang, Minjie Zhang, Fenghui RenAAAI 2018 · DOI

Predicting multiple future steps in a time series—such as electricity demand for the next 24 hours—presents a dual challenge: capturing complex nonlinear relationships between inputs and outputs, and modeling the inherent dependencies between the output time steps themselves. Traditional linear models like ARIMA often fail to capture nonlinearities, while deep learning models like Recurrent Neural Networks (RNNs) frequently treat output steps as independent, neglecting the structured nature of the sequence.

To bridge this gap, researchers presented a hybrid architecture called CoR (Conditional Random Fields on top of Recurrent Neural Networks) at AAAI-18. This model integrates Sparse Gaussian Conditional Random Fields (SGCRF) with deep RNNs to create a system that is both a powerful feature extractor and a sophisticated reasoner about output dependencies.

The Challenge of Multi-Step Forecasting

Multi-step forecasting is difficult because of two primary factors:

  1. Nonlinear Input Correlations: The relationship between observed features (e.g., weather, historical usage) and the target is often highly nonlinear and temporally dependent.
  2. Structured Output Dependencies: Predicted values are not independent; for instance, energy demand at 10:00 AM is strongly correlated with demand at 11:00 AM.

While RNNs excel at the first challenge, they often struggle to explicitly model the "structure" of the entire output sequence. Conversely, SGCRFs are excellent at modeling output structure but typically require extensive manual feature engineering to handle raw, nonlinear input data.

The CoR Architecture

The CoR architecture consists of two primary modules stacked vertically to leverage the strengths of both deep learning and probabilistic graphical models.

The CoR architecture: Stacked Bi-RNNs serve as a feature extractor for a Sparse Gaussian CRF (SGCRF) structured head.
  • The Feature Extractor (Bottom): Stacked Bi-directional RNNs (or LSTMs) process the input sequence $X$ to produce an intermediate representation $z$. This automated feature extraction replaces the need for manual engineering, allowing the model to learn complex temporal patterns directly from raw data.
  • The Structured Head (Top): An SGCRF takes the representation $z$ and predicts the final sequence $y$. It utilizes a precision matrix ($\Lambda$) to explicitly model the conditional dependencies between different time steps in the output sequence.

Joint Optimization and Training

Training this hybrid model is complex because the precision matrix in a Gaussian CRF must remain positive-definite. The authors found that end-to-end training using Projected Gradient Descent (PGD) was the most effective approach. During backpropagation, if the precision matrix becomes invalid, it is "projected" back onto the space of positive-definite matrices using eigenvalue decomposition. This joint optimization allows the RNN to learn representations specifically tailored for the structured reasoning of the CRF.

Experimental Evaluation and Results

The model was rigorously tested on synthetic datasets and the real-world NPower Forecasting Challenge 2016 dataset. In the NPower Challenge, the best CoR variant (using LSTMs and end-to-end training) achieved a Mean Absolute Percentage Error (MAPE) of 3.87%, significantly outperforming the competition's first-place benchmark of 7.13%.

Beyond accuracy, CoR offers substantial computational advantages. Pure SGCRFs struggle with high-dimensional features because their calculations scale quadratically. By using the RNN to compress inputs into a smaller intermediate state, CoR maintains efficiency even with large-scale data.

Performance comparison showing CoR's improvements in accuracy (MAPE) and computational efficiency over standalone models.

As shown in the table above, for long sequences ($T=200$), CoR is not only more accurate than a pure SGCRF but also significantly faster and more memory-efficient. Specifically, it demonstrated a 5x speedup and used approximately 80% less memory compared to the standalone SGCRF.

Conclusion

The CoR framework demonstrates that combining deep recurrent networks with structured graphical models provides a robust solution for complex time-series forecasting. By automating feature engineering through RNNs and enforcing output consistency through SGCRFs, the model achieves state-of-the-art performance with improved scalability. Future work may explore adapting this "RNN + Structured Head" approach to other domains like natural language processing or computer vision, where structured output is equally critical.

Summary updated

BibTeX
@inproceedings{wang2018sparse,
  author = {Xishun Wang and Minjie Zhang and Fenghui Ren},
  title = {{Sparse Gaussian Conditional Random Fields on Top of Recurrent Neural Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.11633},
}
Figure