Introduction
Long-term time-series forecasting (LTTF) is a cornerstone of modern industrial planning, essential for sectors ranging from energy grid management to global transportation logistics. While the Transformer architecture has revolutionized sequence modeling, its application to long-term forecasting has been hampered by computational bottlenecks and an inability to fully capture the nuanced dependencies inherent in temporal data. This paper introduces Conformer, a novel Transformer-based model designed to provide high-performance, stable predictions with linear computational complexity.
The Challenges of Long-Term Forecasting
Existing models for LTTF face three primary hurdles that limit their practical utility in real-world scenarios:
- Computational Efficiency: The standard self-attention mechanism scales quadratically ($O(L^2)$) with sequence length. While various "sparse" attention mechanisms have been proposed to mitigate this, they often achieve efficiency by discarding significant portions of the input data, leading to a loss of critical information.
- Complex Dependencies: Real-world time-series data exhibit intricate "vertical" correlations (inter-variable relationships) and "horizontal" multi-scale temporal patterns (e.g., overlapping hourly, daily, and seasonal cycles). Capturing these simultaneously remains a significant architectural challenge.
- Uncertainty Quantification: Most forecasting models are discriminative, providing a single "point estimate." However, time-series data is inherently noisy (aleatoric uncertainty). Without a measure of confidence or a distribution of potential outcomes, these point estimates can be unreliable for high-stakes decision-making.
The Conformer Architecture
To address these challenges, the Conformer architecture integrates three innovative components designed to optimize feature representation, pattern extraction, and distribution modeling.
Feature-Level Representation via FFT
Conformer utilizes the Fast Fourier Transform (FFT) to explicitly model multivariate correlations in the frequency domain. By shifting the perspective from the time domain to the frequency domain, the model can more effectively identify inter-variable relationships that might be obscured by noise. Additionally, the model extracts multi-scale dynamics by embedding data at various temporal resolutions, such as minutes, hours, and days.
Pattern Modeling: Sliding-Window Attention and SIRN
To maintain linear complexity ($O(L)$), Conformer employs sliding-window attention, which restricts the attention mechanism to local neighbors. To compensate for the resulting loss of global context, the authors introduce the Stationary and Instant Recurrent Network (SIRN). SIRN utilizes a recurrent structure to decompose signals into "stationary" (long-term trends) and "instant" (seasonal or local) patterns. This hybrid approach ensures that the model captures both local fluctuations and global trends without the quadratic cost of full-sequence attention.
Distribution Modeling with Normalizing Flows
Rather than outputting a simple scalar value, Conformer incorporates a Normalizing Flow (NF) block. This block uses a chain of invertible transformations to map a simple base distribution to the complex, multi-modal distribution of the target series. This generative approach allows the model to be more robust to noise and provides reliable uncertainty quantification, offering a range of probable outcomes rather than a single, potentially fragile prediction.
Experimental Evaluation
The Conformer model was rigorously evaluated against seven real-world datasets, including standard benchmarks and newly collected datasets:
- Standard Benchmarks: ECL (Electricity consumption), Weather, Exchange (currency rates), and ETTh1/ETTm1 (Electricity Transformer Temperature).
- New Benchmarks: A Wind power farm dataset (15-minute intervals) and AirDelay (flight arrival delays in Texas). The AirDelay dataset is particularly notable for its irregular time intervals, which test the model's robustness to less-structured data.
Key Results and Insights
State-of-the-Art Performance
Conformer consistently outperformed established baselines, including Autoformer, Informer, and LogTrans. The performance gains were particularly striking on specific datasets: a 41% reduction in Mean Squared Error (MSE) on the Exchange dataset and a 33.5% reduction on the ECL dataset compared to the next-best performing models.
Horizon Stability
A critical metric for LTTF is how error scales as the prediction window increases. As the horizon extended from 96 to 768 time steps, Conformer’s error rate grew significantly slower than its competitors. This stability suggests that the combination of SIRN and sliding-window attention is highly effective at preserving long-term dependencies.
Efficiency and Robustness
By maintaining linear time and memory complexity, Conformer proves practical for processing the long sequences required for modern forecasting tasks where standard Transformers often exceed memory limits. Furthermore, the use of Normalizing Flows demonstrated that generative modeling can effectively "absorb" data volatility, leading to more stable long-term predictions.
Conclusion
Conformer represents a significant step forward in long-term time-series forecasting by harmonizing the efficiency of linear-complexity models with the deep representational power of Transformers. By addressing feature correlation, pattern decomposition, and distribution modeling in a single architecture, it provides a robust framework for complex forecasting tasks. Future work will focus on further optimizing the computational overhead of the Normalizing Flow and SIRN modules and exploring specialized architectures for even more irregular, less-structured temporal data.
Summary updated