Introduction
Long Sequence Time-Series Forecasting (LSTF) is a cornerstone of modern predictive analytics, with applications ranging from energy grid optimization to financial market modeling. Despite the success of Transformer architectures in sequence modeling, their application to time-series data has been hindered by two primary challenges: the "semantic gap" and the "lookback paradox." The semantic gap refers to the lack of inherent meaning in individual scalar time steps, while the lookback paradox describes the phenomenon where increasing historical context (the lookback window) fails to improve, or even degrades, model performance.
Skip-Timeformer addresses these issues by rethinking how Transformers process temporal data. Instead of point-by-point modeling, it introduces a multi-scale approach that captures periodic dependencies and long-term trends through a novel skip-time interaction mechanism.
Methodology
The Skip-Timeformer architecture is built upon three core innovations designed to extract meaningful temporal features from raw sequences.
Multi-skip Sequence Token Embedding
Traditional Transformers treat each time step as an independent token, which often obscures broader periodic patterns. Skip-Timeformer partitions multivariate series into multiple subsequences using a skip step ($\mu$). This process effectively samples the data at regular intervals, creating tokens that represent fragments of the overall trend across different time scales.
Skip-Time Interaction and STICLN
The model applies Multi-head Self-Attention (MSA) directly to these sampled subsequences. This "Skip-Time Interaction" allows the model to learn dependencies between distant points that share periodic relationships. To integrate this multi-scale information back into the primary sequence processing path, the authors introduce Skip-Time Interaction Conditional Layer Normalization (STICLN). STICLN dynamically updates normalization parameters ($\alpha$ and $\beta$) based on the sampled features, ensuring that global periodic information informs local sequence processing.
Dynamic Dropout
To mitigate overfitting and handle noise in long sequences, Skip-Timeformer employs a progressive dynamic dropout scheme. By introducing Bernoulli noise during training, the model is forced to learn more robust and generalizable temporal representations.
Experimental Evaluation
Skip-Timeformer was rigorously tested against eight real-world benchmarks, including the Electricity Transformer Temperature (ETT) datasets, Weather, Electricity, Exchange, and Traffic.
Performance and Generality
The model consistently achieved state-of-the-art performance, yielding the lowest Mean Squared Error (MSE) and Mean Absolute Error (MAE) across the board. Notably, Skip-Timeformer outperformed PatchTST, particularly in datasets characterized by high-frequency fluctuations.
A key strength of the Skip-Time framework is its generality. When applied to other Transformer variants such as Informer, Flowformer, and FlashAttention, it produced average performance improvements ranging from 28% to 57%. For instance, on the ETTh1 dataset, the framework reduced the MSE of a standard Transformer from 1.003 to 0.427.
Breaking the Lookback Limit
Unlike previous models that plateaued with more data, Skip-Timeformer’s accuracy continues to scale with the size of the lookback window. This suggests that the skip-sampling approach allows the model to effectively "digest" longer historical contexts without being overwhelmed by local noise.
Conclusion
Skip-Timeformer represents a significant advancement in LSTF by demonstrating that the limitations of Transformers in time-series forecasting are not inherent to the architecture, but rather a result of suboptimal data representation and normalization. By leveraging skip-time interactions, the model bridges the semantic gap and resolves the lookback paradox, setting a new standard for long-term forecasting accuracy and efficiency.
Summary updated