Correlated Time Series Forecasting using Deep Neural Networks: A Summary of Results

Razvan-Gabriel Cirstea, Darius-Valer Micu, Gabriel-Marcel Muresan, Chenjuan Guo, Bin YangarXiv 2018 · arXiv

Introduction

Forecasting correlated time series is a critical task in managing complex cyber-physical systems, such as urban sewage networks or traffic systems. In these environments, sensors record data that is inherently interlinked; for instance, the concentration of one chemical in a treatment plant often influences another due to underlying biological processes. Traditional forecasting methods often struggle with these dynamics due to linearity constraints and a lack of robust feature extraction.

Problem Statement

Standard forecasting approaches face two primary hurdles. First, linear models like ARIMA cannot adequately represent the complex, non-linear interactions within cyber-physical systems. Second, while deep learning models like LSTMs can handle non-linearity, they often process raw, noisy inputs directly. Without specialized feature extraction, these models can overfit to noise or fail to capture the subtle correlations between different series. The objective is to develop a framework that performs automated feature engineering on individual series while simultaneously accounting for their cross-correlations over time.

Proposed Architectures

The authors propose a modular approach combining Convolutional Neural Networks (CNNs) for spatial/local feature extraction and Recurrent Neural Networks (RNNs) for temporal modeling. Two novel architectures are introduced:

1. CRNN (Convolutional Recurrent Neural Network)

In the CRNN model, each individual time series is first fed into its own 1D-CNN. This allows the model to learn "local" patterns and features specific to that variable. These extracted features are then merged into a single vector and passed to an RNN layer to learn sequential dependencies and correlations between the different series.

2. AECRNN (Auto-Encoder CRNN)

The AECRNN builds on the CRNN by adding a deconvolutional (decoder) branch, transforming the model into a multi-task learner.

The AECRNN architecture integrates a deconvolutional layer for input reconstruction, facilitating robust feature extraction through multi-task learning.

As illustrated in the AECRNN architecture, the model is trained to minimize a joint loss function consisting of forecasting error and reconstruction error. This auto-encoding task acts as a regularizer, forcing the CNN layers to learn representative, "clean" features of the input by filtering out uncorrelated noise and sensor glitches.

Experimental Evaluation

The models were evaluated on two distinct datasets:

  • Sewage Treatment Data: A real-world dataset from Aalborg, Denmark, containing three years of measurements for $NH_4$, $NO_3$, and $O_2$ across six different tanks, sampled every 2 minutes.
  • Google Trends: Monthly search popularity for the keywords "potatoes" and "brown sugar" over 14 years, chosen for their high correlation.

Results and Discussion

Experimental results demonstrated that both CRNN and AECRNN consistently outperformed baselines like LSTM, RNN, and ARIMA in terms of Root Mean Square Error (RMSE) and Mean Absolute Percentage Error (MAPE). While simple linear models were effective for 1-step-ahead forecasting, their performance deteriorated rapidly as the prediction window ($p$) increased, whereas the proposed deep learning models showed significant strength in long-term forecasting.

Robustness and Interpretability

A key finding was the superior robustness of AECRNN. In experiments with uncorrelated "dummy" noise, AECRNN’s error stayed relatively stable compared to the standard CRNN. Furthermore, visual analysis of the internal "pooling" layers revealed that AECRNN’s filters produced more human-readable representations of the data trends.

Comparison of the internal representations (pooling layer outputs) for AECRNN and CRNN. AECRNN produces smoother, more interpretable feature trends.

As shown in the comparison of forecasting layer inputs, the AECRNN learns smoother transitions and clearer trends, whereas the standard CRNN output appears more chaotic. This suggests that the multi-task objective leads to a more physically meaningful internal state, effectively separating signal from noise.

Conclusion

The CRNN and AECRNN architectures provide a robust framework for forecasting correlated time series. By decoupling feature extraction from temporal modeling and utilizing multi-task learning, these models achieve high accuracy and noise resilience. Future work will investigate the scalability of these architectures to larger sensor networks and their application in real-time transportation network forecasting.

Summary updated

Abstract

Cyber-physical systems often consist of entities that interact with each other over time. Meanwhile, as part of the continued digitization of industrial processes, various sensor technologies are deployed that enable us to record time-varying attributes (a.k.a., time series) of such entities, thus producing correlated time series. To enable accurate forecasting on such correlated time series, this paper proposes two models that combine convolutional neural networks (CNNs) and recurrent neural networks (RNNs). The first model employs a CNN on each individual time series, combines the convoluted features, and then applies an RNN on top of the convoluted features in the end to enable forecasting. The second model adds additional auto-encoders into the individual CNNs, making the second model a multi-task learning model, which provides accurate and robust forecasting. Experiments on two real-world correlated time series data set suggest that the proposed two models are effective and outperform baselines in most settings. This report extends the paper "Correlated Time Series Forecasting using Multi-Task Deep Neural Networks," to appear in ACM CIKM 2018, by providing additional experimental results.
BibTeX
@misc{cirstea2018correlated,
  author = {Razvan-Gabriel Cirstea and Darius-Valer Micu and Gabriel-Marcel Muresan and Chenjuan Guo and Bin Yang},
  title = {{Correlated Time Series Forecasting using Deep Neural Networks: A Summary of Results}},
  year = {2018},
  eprint = {1808.09794},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.09794},
}
Figure