Tensorized LSTM with Adaptive Shared Memory for Learning Trends in Multivariate Time Series

Dongkuan Xu, Wei Cheng, Bo Zong, Dongjin Song, Jingchao Ni, Wenchao Yu, Yanchi Liu, Haifeng Chen, Xiang Zhang

Introduction

Traditional time series forecasting primarily focuses on point prediction—estimating the exact value of a variable at a future time step. However, in many industrial and financial applications, such as capacity planning or algorithmic trading, the underlying trend (characterized by slope and duration) is often more critical and less susceptible to high-frequency noise. Predicting these trends in multivariate settings presents significant challenges: temporal and spatial patterns are often deeply entangled, and the relationships between different time series can be highly dynamic.

Methodology: The DeepTrends Architecture

To address these challenges, we propose DeepTrends, a novel neural network architecture designed for multivariate trend forecasting. DeepTrends employs a dual-pathway approach to capture both long-term dependencies and short-term local fluctuations.

Tensorized LSTM with Adaptive Shared Memory (TLASM)

The core of DeepTrends is the TLASM module. Unlike standard LSTMs that collapse all variables into a single hidden state vector, TLASM represents the hidden state as a matrix (tensor). This "tensorization" preserves the individual dynamics of each time series. Furthermore, an Adaptive Shared Memory mechanism is introduced. Instead of static parameter sharing, an attention-based router dynamically determines how much information should be shared between different time series at each time step, effectively mitigating "negative transfer" between unrelated tasks.

Multi-task 1dCNN and Feature Fusion

While the TLASM handles long-term trend sequences, a parallel Multi-task 1dCNN extracts local features from raw data to account for sudden, short-term shifts. Finally, task-specific Multi-Layer Perceptrons (MLPs) fuse these long-term and short-term features to generate the final predictions for slope and duration.

The DeepTrends architecture overview, showcasing the integration of TLASM blocks and the multi-task 1dCNN for trend forecasting.

Experimental Evaluation

The model was evaluated on three diverse multivariate datasets: Traffic (hourly road occupancy), Exchange-Rate (daily currency rates), and Solar-Power (high-frequency production records). DeepTrends was compared against seven baseline models, including standard recurrent and convolutional architectures, as well as TreNet, the previous state-of-the-art for univariate trend prediction.

Results and Discussion

Experimental results demonstrate that DeepTrends consistently outperforms all baselines, achieving the lowest Root Mean Square Error (RMSE) and Root Relative Squared Error (RRSE) across all datasets for both slope and duration prediction.

Visualization of trend prediction performance, comparing true vs. predicted slopes across different time series.

Ablation studies confirmed the necessity of each component: removing either the tensorized states or the local 1dCNN led to a significant drop in performance. Visual analysis of specific cases showed that DeepTrends successfully identifies trend reversals by leveraging long-term dependencies that simpler models overlook.

Conclusion

DeepTrends introduces a robust framework for multivariate trend forecasting by combining tensorized hidden states with adaptive information sharing. While the model exhibits superior accuracy, future work will address its increased computational complexity and explore its scalability to datasets with thousands of concurrent time series.

Summary updated

Abstract

The problem of learning and forecasting underlying trends in time series data arises in a variety of applications, such as traffic management, energy optimization, etc. In literature, a trend in time series is characterized by the slope and duration, and its prediction is then to forecast the two values of the subsequent trend given historical data of the time series. For this problem, existing approaches mainly deal with the case in univariate time series. However, in many real-world applications, there are multiple variables at play, and handling all of them at the same time is crucial for an accurate prediction. A natural way is to employ multi-task learning (MTL) techniques in which the trend learning of each time series is treated as a task. The key point of MTL is to learn task relatedness to achieve better parameter sharing, which however is challenging in trend prediction task. First, effectively modeling the complex temporal patterns in different tasks is hard as the temporal and spatial dimensions are entangled. Second, the relatedness among tasks may change over time. In this paper, we propose a neural network, DeepTrends, for multivariate time series trend prediction. The core module of DeepTrends is a tensorized LSTM with adaptive shared memory (TLASM). TLASM employs the tensorized LSTM to model the temporal patterns of long-term trend sequences in an MTL setting. With an adaptive shared memory, TLASM is able to learn the relatedness among tasks adaptively, based upon which it can dynamically vary degrees of parameter sharing among tasks. To further consider short-term patterns, DeepTrends utilizes a multi-task 1dCNN to learn the local time series features, and employs a task-specific sub-network to learn a mixture of long-term and short-term patterns for trend prediction. Extensive experiments on real datasets demonstrate the effectiveness of the proposed model.
BibTeX
@inproceedings{xu2020tensorized,
  author = {Dongkuan Xu and Wei Cheng and Bo Zong and Dongjin Song and Jingchao Ni and Wenchao Yu and Yanchi Liu and Haifeng Chen and Xiang Zhang},
  title = {{Tensorized LSTM with Adaptive Shared Memory for Learning Trends in Multivariate Time Series}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i02.5496},
}
Figure