Revisiting Attention for Multivariate Time Series Forecasting

Haixiang WuAAAI 2025 · DOI

TL;DR

This paper challenges the conventional attention mechanism in Transformer-based multivariate time series forecasting (MTSF), arguing it might not be optimal. They introduce two novel attention mechanisms, FSatten and SOatten, which leverage frequency domain and learnable orthogonal spaces respectively, achieving significant performance improvements over state-of-the-art methods.

What Problem Does This Paper Address?

Multivariate Time Series Forecasting (MTSF) is crucial for many real-world applications like finance and electricity. Transformer models have shown great success in MTSF by using attention mechanisms to capture dependencies between time series. However, these models typically map time series data into a latent space using linear projections for queries, keys, and values before computing attention. The authors question whether this learned latent space is truly optimal for MTSF and explore alternative approaches to better capture complex dependencies within time series data.

Methods

The paper proposes two novel attention mechanisms:

  • Frequency Spectrum Attention (FSatten): This method shifts the focus to the frequency domain. It uses the Fast Fourier Transform (FFT) to embed the time series data, allowing it to capture periodic dependencies. Instead of linear projections for queries and keys, FSatten introduces Multi-head Spectrum Scaling (MSS), which scales the amplitude of different frequency components. This aims to better identify and align correlated frequency patterns across different time series.
  • Scaled Orthogonal Attention (SOatten): Recognizing that a fixed frequency domain might not suit all scenarios, SOatten offers a more general approach. It creates a learnable orthogonal latent space. A key component is the Head Coupling Convolution (HCC) module, which uses the similarity between adjacent sequences to guide the learning of this orthogonal space, aiming for more comprehensive dependency patterns.

Datasets

The proposed methods were evaluated on six real-world long-term forecasting benchmarks:

  • ECL
  • ETT (four subsets: ETTm1, ETTm2, ETTh1, ETTh2)
  • Exchange
  • Traffic
  • Weather
  • Solar-Energy

Key Insights and Ideas

  • The latent space matters: The paper strongly suggests that the standard linear projection for attention in MTSF might be a bottleneck, and exploring alternative mapping spaces can yield significant improvements.
  • Frequency domain is powerful for periodic data: FSatten's success on datasets with clear periodicity highlights the benefit of explicitly modeling frequency components.
  • Generalization is key: SOatten's improved performance across diverse datasets indicates that a more flexible, learnable orthogonal space, guided by neighboring similarity, offers better generalization than a fixed frequency approach.
  • Neighboring similarity is crucial: The HCC module's effectiveness in SOatten underscores the importance of leveraging relationships between adjacent sequences to learn robust attention patterns.
  • Orthogonal spaces offer stability: Visualizations and numerical analysis suggest that the orthogonal spaces learned by FSatten and SOatten are more informative and stable against noise compared to the conventional attention's latent space.

Key Results

  • FSatten and SOatten consistently outperform state-of-the-art (SOTA) methods that use conventional attention.
  • On average, FSatten achieved an 8.1% improvement over SOTA on MSE for long-term MTSF.
  • SOatten demonstrated even more significant improvements, achieving an average of 21.8% on MSE across datasets.
  • Ablation studies confirmed the effectiveness of both the MSS module (outperforming linear mapping) and the HCC module (significantly enhancing SOatten's performance, especially for Temporal Transformers).
  • Visualizations showed that FSatten and SOatten generate more refined attention weight maps, capturing complex associations and dependencies more effectively.

Limitations and Future Work

  • Dataset Variance: While FSatten performs well on periodic data, its performance can vary across datasets. SOatten offers better generalization but might still face challenges with datasets having a very large number of variables (e.g., Traffic).
  • Future Directions: The authors suggest exploring modern state-space models like Mamba to handle scenarios with a large number of variables more effectively. They also aim to further leverage periodicity and learnable characteristics to compress large variable backgrounds and retain essential information.

Summary updated

BibTeX
@inproceedings{wu2025revisiting,
  author = {Haixiang Wu},
  title = {{Revisiting Attention for Multivariate Time Series Forecasting}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i20.35455},
}
Figure