Introduction
While decoder-only Transformers have revolutionized natural language processing, they have historically struggled in Time Series Forecasting (TSF). Two primary challenges hinder their adoption: the "error accumulation" problem, where recursive single-step predictions lead to cascading inaccuracies, and the difficulty of simultaneously modeling long-range seasonal patterns alongside local temporal fluctuations. This paper introduces WAVE (Weighted Autoregressive Varying gatE), a novel attention mechanism that integrates classical Autoregressive Moving-average (ARMA) logic into the Transformer architecture to address these limitations.
Methodology
Patch-based Tokenization
To mitigate error accumulation, WAVE employs a patch-based approach. Rather than predicting individual time points, the model treats a segment of data—representing the entire forecast horizon—as a single token. This allows the decoder-only model to perform a direct, one-step prediction for the future window, bypassing the pitfalls of autoregressive step-by-step generation.
WAVE Attention Mechanism
The core innovation is the WAVE attention mechanism, inspired by the Box-Jenkins ARMA model. It decouples the temporal modeling into two components:
- Autoregressive (AR) Term: Captures stable, long-range, and cyclic patterns (seasonality).
- Moving-average (MA) Term: Captures short-term fluctuations and prediction errors (shocks).
To maintain computational efficiency, the authors developed an "indirect" MA weight generation method using a linear RNN structure. This ensures the mechanism operates with $O(N)$ time complexity, making it as efficient as standard linear attention while providing superior modeling capabilities. Additionally, the use of LeakyReLU activations for queries allows for negative weights, which are essential for the smoothing effects characteristic of moving-average models.
Experimental Results
WAVE was evaluated across 12 diverse real-world TSF benchmarks, including weather, electricity consumption (ECL), and traffic data (PEMS). The results demonstrate that WAVE consistently outperforms pure AR attention models and established baselines like PatchTST, iTransformer, and DLinear.
In a comprehensive ranking of 48 sub-experiments, WAVE models—particularly the Gated Linear Attention + ARMA variant—achieved an average rank of 2.333, significantly better than the baseline models. Furthermore, WAVE demonstrated robust scalability; unlike many TSF models that overfit with larger lookback windows, WAVE's performance improved as the historical context increased from 512 to 4096 points.
Discussion and Conclusion
WAVE represents a successful synthesis of classical statistical insights and modern deep learning architectures. By introducing the MA component, the model can "absorb" local noise, allowing the AR component to focus on long-term dependencies. While current implementations are channel-independent, future work will explore multivariate extensions to capture inter-series relationships. WAVE proves that with appropriate tokenization and attention design, decoder-only Transformers are highly competitive candidates for state-of-the-art time series forecasting.
Summary updated