Introduction
In long-term time series forecasting (LTSF), the "lookback window"—the extent of historical data available to a model—is a critical hyperparameter. Intuitively, longer historical contexts should enable models to better capture long-term trends and seasonal cycles. However, state-of-the-art models such as Transformers and CNNs frequently encounter a performance ceiling or computational barrier when lookback windows extend into "ultra-long" territory (e.g., 3000+ time steps).
The authors identify three primary challenges hindering the use of ultra-long windows:
- Overfitting: Extended windows often introduce more noise than signal, leading models to memorize historical anomalies rather than learning generalizable patterns.
- Computational Constraints: Many architectures, particularly those based on attention mechanisms, suffer from quadratic memory complexity. This often results in "Out of Memory" (OOM) errors during training.
- Information Extraction: Distilling relevant predictive patterns from massive historical datasets remains difficult for standard architectures.
Methodology: The IRPA Framework
To address these issues, the authors propose IRPA (Input Refinement and Prediction Auxiliary), a lightweight, "plug-and-play" framework consisting of only four linear layers. IRPA is designed to augment existing forecasting models by effectively leveraging ultra-long history without the associated computational overhead.
The framework operates through two core components:
- Input Refinement Module (IRM): Rather than processing the entire ultra-long series through a heavy backbone model, the IRM uses the long history to "refine" a shorter, manageable input window. It decomposes the series into seasonal and trend components and identifies the most similar historical patches (using Hellinger distance) to enrich the current input.
- Prediction Auxiliary Module (PAM): This module provides auxiliary guidance by searching the ultra-long history for patches similar to the current context and observing subsequent historical outcomes. This "future-from-the-past" information serves as a predictive hint for the final forecast.
By relying on linear layers and efficient similarity searches, IRPA maintains a computational complexity of $O(l)$, where $l$ is the lookback length, making it significantly more efficient than attention-based alternatives.
Experimental Evaluation
The framework was evaluated on eight widely recognized public benchmarks, including the ETT (Electricity Transformer Temperature) datasets, Weather, Solar, ECL (Electricity Consumption Load), and Traffic. These datasets vary significantly in feature count and sampling frequency, providing a robust testing ground for the framework's versatility.
Key Results and Insights
IRPA demonstrated significant performance improvements across all tested baseline models. On average, it reduced Mean Squared Error (MSE) by 9.7% for iTransformer, 15.0% for PatchTST, 14.4% for TimesNet, and 25.2% for the DLinear model.
A key finding is that the backbone model does not need to process the entire ultra-long window directly. Instead, the ultra-long window serves as a reference library to augment a smaller input. Furthermore, the research highlights a "Frequency-Window Scaling" principle: data with higher sampling frequencies requires proportionally longer lookback windows to achieve optimal forecasting accuracy.
In terms of efficiency, IRPA-enhanced models successfully avoided OOM errors that plagued standard architectures at high lookback lengths. For instance, on the Weather dataset, while a standard PatchTST model failed at a lookback window of 3600, the IRPA-enhanced version maintained a low memory footprint while achieving superior accuracy.
Conclusion and Future Work
IRPA provides a robust solution for scaling time-series models to ultra-long historical contexts. While the optimal lookback window still varies by dataset, the framework's linear efficiency makes it a viable candidate for integration into large-scale time-series foundation models. Future research may explore automated window selection or the use of lightweight non-linear components to further enhance information extraction.
Summary updated