cs-net: structural approach to time-series forecasting for high-dimensional feature space data with limited observations

Weiyu Zong, Mingqian Feng, Griffin Heyrich, Peter ChinarXiv 2022 · OpenReview

Abstract

In recent years, deep-learning-based approaches have been introduced to solving time-series forecasting-related problems. These novel methods have demonstrated impressive performance in univariate and low-dimensional multivariate time-series forecasting tasks. However, when these novel methods are used to handle high-dimensional multivariate forecasting problems, their performance is highly restricted by a practical training time and a reasonable GPU memory configuration. In this paper, inspired by a change of basis in the Hilbert space, we propose a flexible data feature extraction technique that excels in high-dimensional multivariate forecasting tasks. Our approach was originally developed for the National Science Foundation (NSF) Algorithms for Threat Detection (ATD) 2022 Challenge. Implemented using the attention mechanism and Convolutional Neural Networks (CNN) architecture, our method demonstrates great performance and compatibility. Our models trained on the GDELT Dataset finished 1st and 2nd places in the ATD sprint series and hold promise for other datasets for time series forecasting.

Introduction

Most modern deep learning models for time-series forecasting—such as LSTMs or Transformers—excel when they have massive amounts of historical data and a relatively small number of variables. However, real-world problems often present a different challenge: high-dimensional multivariate data with limited historical observations. For instance, a dataset might contain hundreds of geographic regions and dozens of event types, but only a few hundred weeks of history.

In these "high-dimensional, limited observation" scenarios, standard models face two major hurdles: prohibitive computational costs due to massive GPU memory requirements and a high risk of overfitting. The authors developed CS-Net (Cross-Sectional Net) to bridge this gap, originally for the NSF Algorithms for Threat Detection (ATD) 2022 Challenge, where the goal was to predict global conflict events using the sparse and complex GDELT dataset.

Methodology

The core innovation of CS-Net is its "structural approach," which treats time-series data more like an image than a flat list of numbers. Instead of flattening data into a table, the framework arranges it into a 3D cuboid with dimensions for Time, Event Type, and Region, preserving the natural geometry of the data.

Model Architecture

The methodology consists of several key pillars integrated into a unified pipeline:

  • Attention-Based Encoder: To capture relationships between different geographic locations, the model uses a multi-head self-attention layer. This helps the model understand how an event in one region might influence another.
  • Convolution-Based Signal Transform: Inspired by image processing, the model applies 2D convolutions to "slices" of the data cuboid. This acts as a change of basis in the Hilbert space, allowing the model to extract complex features much faster than recurrent networks.
  • Model Ensemble: To balance bias and variance, CS-Net ensembles the neural network output with a classical Vector Auto-Regression (VAR) model. While deep learning excels at catching sudden changes, the VAR model is better at following general trends.
The CS-Net architecture integrates multi-head self-attention and 2D convolutions with a final ensemble layer.

Experimental Setup

The researchers evaluated CS-Net on two challenging datasets:

  1. GDELT (ATD 2022 Challenge): A dataset tracking 20 different types of "Conflict and Mediation" events across 260 geographic regions over 215 weeks. This is a highly sparse, high-dimensional dataset.
  2. Wikipedia Web Traffic: Daily traffic data for 1,400 web pages over 500 days, including structural dimensions like User Device, Language, and Keyword categories.

Key Results

CS-Net 3 (the full version of the model) consistently outperformed top-tier baselines including Amazon's DeepAR, Google's Temporal Fusion Transformer (TFT), and N-Beats.

Performance comparison of CS-Net variants against baseline models on ATD and Wikipedia datasets.

As shown in the table above, on the GDELT dataset, CS-Net achieved a Mean Absolute Scaled Error (MASE) of 1.015, outperforming TFT (1.078) and DeepAR (1.136). The model also showed a significant lead in Mean Squared Error (MSE), suggesting it is particularly effective at avoiding large, catastrophic prediction errors. This approach successfully took 1st and 2nd places in the ATD sprint series.

Conclusion

CS-Net demonstrates that high dimensionality, often seen as a "curse," can be leveraged as an advantage if the data's structure is exploited. By treating cross-sections of multivariate data as 2D panels, the authors could leverage efficient 2D convolutions. While the model currently relies on clear categorical structures and offers less interpretability than models like TFT, its performance in data-sparse environments marks a significant advancement in forecasting research. Future work will explore providing more historical observations, which is expected to further increase the performance gap between CS-Net and its competitors.

Summary updated

Abstract

In recent years, deep-learning-based approaches have been introduced to solving time-series forecasting-related problems. These novel methods have demonstrated impressive performance in univariate and low-dimensional multivariate time-series forecasting tasks. However, when these novel methods are used to handle high-dimensional multivariate forecasting problems, their performance is highly restricted by a practical training time and a reasonable GPU memory configuration. In this paper, inspired by a change of basis in the Hilbert space, we propose a flexible data feature extraction technique that excels in high-dimensional multivariate forecasting tasks. Our approach was originally developed for the National Science Foundation (NSF) Algorithms for Threat Detection (ATD) 2022 Challenge. Implemented using the attention mechanism and Convolutional Neural Networks (CNN) architecture, our method demonstrates great performance and compatibility. Our models trained on the GDELT Dataset finished 1st and 2nd places in the ATD sprint series and hold promise for other datasets for time series forecasting.
BibTeX
@misc{zong2022cs,
  author = {Weiyu Zong and Mingqian Feng and Griffin Heyrich and Peter Chin},
  title = {{cs-net: structural approach to time-series forecasting for high-dimensional feature space data with limited observations}},
  year = {2022},
  eprint = {2212.02567},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.02567},
}
Figure