The Power of Minimalism in Long Sequence Time-series Forecasting

Zhao Sun, Meng Fang, Dongjin Song, Defu Li, Tianjin Huang, Qinke PengOpenReview 2024 · Submission to ICLR 2024 · Rejected · OpenReview

Abstract

Recently, transformer-based models have been widely applied to time series forecasting tasks due to their remarkable capability to capture complex interactions within sequential data. However, as the sequence length expands, Transformer-based models suffer from increased memory consumption, overfitting, and performance deterioration in capturing long-range dependencies. Recently, several studies have shown that MLP-based models can outperform advanced Transformer-based models for long-term time series forecasting (LTSF) tasks. Unfortunately, linear mappings often struggle to capture intricate dependencies when handling multivariate time series. Although modeling each channel independently can alleviate this issue, it will significantly increase the computational cost. To this end, we introduce a set of simple yet effective depthwise convolution models named LTSF-Conv to perform LTSF. Specifically, we apply unique filters to each channel to achieve channel independence, which plays a pivotal role in enhancing overall forecasting performance. Experimental results show that LTSF-Conv models outperform the state-of-the-art Transformer-based and MLP-based models across seven real-world LTSF benchmarks. Surprisingly, a two-layer non-stacked network can outperform the state-of-the-art Transformer model in 91\% of cases with a significant reduction of computing resources. In particular, LTSF-Conv models substantially decrease the average number of trainable parameters (by $\sim$ 12$\times$), maximum memory consumption (by $\sim$ 86$\times$), running time (by $\sim$ 18$\times$), and inference time (by $\sim$ 2$\times$) on the Electricity benchmark.
BibTeX
@misc{sun2024power,
  author = {Zhao Sun and Meng Fang and Dongjin Song and Defu Li and Tianjin Huang and Qinke Peng},
  title = {{The Power of Minimalism in Long Sequence Time-series Forecasting}},
  year = {2024},
  url = {https://openreview.net/forum?id=hF8jnnexSB},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2024},
}
Figure