Skip-Timeformer: Skip-Time Interaction Transformer for Long Sequence Time-Series Forecasting

Wenchang Zhang, Hua Wang, Fan ZhangIJCAI 2024 · OpenReview

Introduction

Long Sequence Time-Series Forecasting (LSTF) is a cornerstone of modern predictive analytics, with applications ranging from energy grid optimization to financial market modeling. Despite the success of Transformer architectures in sequence modeling, their application to time-series data has been hindered by two primary challenges: the "semantic gap" and the "lookback paradox." The semantic gap refers to the lack of inherent meaning in individual scalar time steps, while the lookback paradox describes the phenomenon where increasing historical context (the lookback window) fails to improve, or even degrades, model performance.

Skip-Timeformer addresses these issues by rethinking how Transformers process temporal data. Instead of point-by-point modeling, it introduces a multi-scale approach that captures periodic dependencies and long-term trends through a novel skip-time interaction mechanism.

Methodology

The Skip-Timeformer architecture is built upon three core innovations designed to extract meaningful temporal features from raw sequences.

Multi-skip Sequence Token Embedding

Traditional Transformers treat each time step as an independent token, which often obscures broader periodic patterns. Skip-Timeformer partitions multivariate series into multiple subsequences using a skip step ($\mu$). This process effectively samples the data at regular intervals, creating tokens that represent fragments of the overall trend across different time scales.

Multi-skip sequence segmentation process

Skip-Time Interaction and STICLN

The model applies Multi-head Self-Attention (MSA) directly to these sampled subsequences. This "Skip-Time Interaction" allows the model to learn dependencies between distant points that share periodic relationships. To integrate this multi-scale information back into the primary sequence processing path, the authors introduce Skip-Time Interaction Conditional Layer Normalization (STICLN). STICLN dynamically updates normalization parameters ($\alpha$ and $\beta$) based on the sampled features, ensuring that global periodic information informs local sequence processing.

Skip-Timeformer overall architecture and STICLN module

Dynamic Dropout

To mitigate overfitting and handle noise in long sequences, Skip-Timeformer employs a progressive dynamic dropout scheme. By introducing Bernoulli noise during training, the model is forced to learn more robust and generalizable temporal representations.

Experimental Evaluation

Skip-Timeformer was rigorously tested against eight real-world benchmarks, including the Electricity Transformer Temperature (ETT) datasets, Weather, Electricity, Exchange, and Traffic.

Performance and Generality

The model consistently achieved state-of-the-art performance, yielding the lowest Mean Squared Error (MSE) and Mean Absolute Error (MAE) across the board. Notably, Skip-Timeformer outperformed PatchTST, particularly in datasets characterized by high-frequency fluctuations.

A key strength of the Skip-Time framework is its generality. When applied to other Transformer variants such as Informer, Flowformer, and FlashAttention, it produced average performance improvements ranging from 28% to 57%. For instance, on the ETTh1 dataset, the framework reduced the MSE of a standard Transformer from 1.003 to 0.427.

Breaking the Lookback Limit

Unlike previous models that plateaued with more data, Skip-Timeformer’s accuracy continues to scale with the size of the lookback window. This suggests that the skip-sampling approach allows the model to effectively "digest" longer historical contexts without being overwhelmed by local noise.

Conclusion

Skip-Timeformer represents a significant advancement in LSTF by demonstrating that the limitations of Transformers in time-series forecasting are not inherent to the architecture, but rather a result of suboptimal data representation and normalization. By leveraging skip-time interactions, the model bridges the semantic gap and resolves the lookback paradox, setting a new standard for long-term forecasting accuracy and efficiency.

Summary updated

Abstract

Recent studies have raised questions about the suitability of the Transformer architecture for long sequence time-series forecasting. These forecasting models leverage Transformers to capture dependencies between multiple time steps in a time series, with embedding tokens composed of data from individual time steps. However, challenges arise when applying Transformers to predict long sequences with strong periodicity, leading to performance degradation and increased computational burden. Furthermore, embedding tokens formed one time step at a time may struggle to reveal meaningful information in long sequences, failing to capture correlations between different time steps. In this study, we propose Skip-Timeformer, a Transformer-based model that utilizes a skip-time interaction for long sequence time-series forecasting. Specifically, we decompose the time series into multiple subsequences based on different time intervals, embedding various time steps into variable tokens across multiple sequences. The skip-time interaction mechanism utilizes these variable tokens to capture dependencies in the skip-time dimension. Additionally, skip-time interaction is employed to learn dependencies between sequences missed by multiple skip time steps. The Skip-Timeformer model demonstrates state-of-the-art performance on various real-world datasets, further enhancing the long sequence forecasting capabilities of the Transformer variations and better adapting to arbitrary lookback windows.
BibTeX
@inproceedings{zhang2024skip,
  author = {Wenchang Zhang and Hua Wang and Fan Zhang},
  title = {{Skip-Timeformer: Skip-Time Interaction Transformer for Long Sequence Time-Series Forecasting}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=KeaRZxJgwP},
}
Figure