Autoregressive Moving-average Attention Mechanism for Time Series Forecasting

Jiecheng Lu, Xu Han, Yan Sun, Shihao Yang

Introduction

While decoder-only Transformers have revolutionized natural language processing, they have historically struggled in Time Series Forecasting (TSF). Two primary challenges hinder their adoption: the "error accumulation" problem, where recursive single-step predictions lead to cascading inaccuracies, and the difficulty of simultaneously modeling long-range seasonal patterns alongside local temporal fluctuations. This paper introduces WAVE (Weighted Autoregressive Varying gatE), a novel attention mechanism that integrates classical Autoregressive Moving-average (ARMA) logic into the Transformer architecture to address these limitations.

Methodology

Patch-based Tokenization

To mitigate error accumulation, WAVE employs a patch-based approach. Rather than predicting individual time points, the model treats a segment of data—representing the entire forecast horizon—as a single token. This allows the decoder-only model to perform a direct, one-step prediction for the future window, bypassing the pitfalls of autoregressive step-by-step generation.

WAVE architecture utilizing patch-based tokenization and RevIN for efficient time series forecasting.

WAVE Attention Mechanism

The core innovation is the WAVE attention mechanism, inspired by the Box-Jenkins ARMA model. It decouples the temporal modeling into two components:

  1. Autoregressive (AR) Term: Captures stable, long-range, and cyclic patterns (seasonality).
  2. Moving-average (MA) Term: Captures short-term fluctuations and prediction errors (shocks).

To maintain computational efficiency, the authors developed an "indirect" MA weight generation method using a linear RNN structure. This ensures the mechanism operates with $O(N)$ time complexity, making it as efficient as standard linear attention while providing superior modeling capabilities. Additionally, the use of LeakyReLU activations for queries allows for negative weights, which are essential for the smoothing effects characteristic of moving-average models.

Experimental Results

WAVE was evaluated across 12 diverse real-world TSF benchmarks, including weather, electricity consumption (ECL), and traffic data (PEMS). The results demonstrate that WAVE consistently outperforms pure AR attention models and established baselines like PatchTST, iTransformer, and DLinear.

Performance rankings across 48 sub-experiments, comparing WAVE variants against standard attention and baseline models.

In a comprehensive ranking of 48 sub-experiments, WAVE models—particularly the Gated Linear Attention + ARMA variant—achieved an average rank of 2.333, significantly better than the baseline models. Furthermore, WAVE demonstrated robust scalability; unlike many TSF models that overfit with larger lookback windows, WAVE's performance improved as the historical context increased from 512 to 4096 points.

Discussion and Conclusion

WAVE represents a successful synthesis of classical statistical insights and modern deep learning architectures. By introducing the MA component, the model can "absorb" local noise, allowing the AR component to focus on long-term dependencies. While current implementations are channel-independent, future work will explore multivariate extensions to capture inter-series relationships. WAVE proves that with appropriate tokenization and attention design, decoder-only Transformers are highly competitive candidates for state-of-the-art time series forecasting.

Summary updated

BibTeX
@inproceedings{lu2025autoregressive,
  author = {Jiecheng Lu and Xu Han and Yan Sun and Shihao Yang},
  title = {{Autoregressive Moving-average Attention Mechanism for Time Series Forecasting}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=Qqn5ktBUxH},
}
Figure