Frequency-Enhanced Transformer with Symmetry-Based Lightweight Multi-Representation for Multivariate Time Series Forecasting

Chenyue Wang, Zhouyuan Zhang, Xin Wang, Mingyang Liu, Lin Chen, Jiatian PiSymmetry 2024 · DOI

TL;DR

This paper introduces a novel transformer-based model for multivariate time series forecasting that leverages frequency domain representations and symmetry to achieve highly accurate predictions with significantly reduced computational cost. By decomposing frequency information into multiple representations and processing them independently, the model captures complex inter-variable dependencies more effectively than existing methods.

What Problem Does This Paper Address?

Multivariate time series forecasting is crucial for many real-world applications, from energy consumption to weather prediction. While transformer models have shown promise, mainstream approaches often struggle with:

  • Inefficient Time-Domain Representations: Key information can be scattered and sparse across time steps, leading to parameter bloat and difficulty in capturing long-term dependencies.
  • Ignoring Inter-Variable Correlations: Treating time steps as uniform tokens can erase the crucial relationships between different variables.
  • High Computational Cost: Standard transformer attention mechanisms have quadratic complexity, making them computationally expensive for long sequences and large datasets.

Existing methods have attempted to address these issues, but often fall short in balancing accuracy, efficiency, and the ability to capture complex multivariate interactions.

Methods

The proposed method, Frequency-Enhanced Transformer with Symmetry-Based Lightweight Multi-Representation (FIR-Attention), tackles these challenges through several key innovations:

  • Frequency Domain Transformation: The time-domain series is first transformed into the frequency domain using Discrete Fourier Transform (DFT).
  • Symmetry-Based Representation: Leveraging the inherent conjugate symmetry in the frequency domain, the model uses only the lower half of the frequency components, significantly reducing dimensionality.
  • Multi-Representation Input: The frequency domain information is decomposed into four distinct representations: amplitude, phase, real part, and imaginary part.
  • Independent Sub-Head Attention (FIR-Attention): Each of these four frequency representations is fed into a separate "sub-head" within a multi-head attention mechanism. This allows each representation to be processed independently, preventing interference and enabling specialized learning of patterns within each frequency component.
  • Lightweight Design:
    • Cutting-Off Frequency: High-frequency noise is removed by retaining only a subset of the lowest frequency components.
  • Equivalent Mapping: The model uses an "equivalent mapping" scheme where the input sequence length is mapped to a target dimension of the same length, optimizing parameter efficiency and reducing spatio-temporal complexity.
  • Complex-Valued Frequency Linear Layer: This layer handles the real and imaginary parts, performing differential mapping and up-sampling to reconstruct the time-domain signal while preserving intricate frequency characteristics.

Datasets

The authors evaluated their model on eight widely used, publicly accessible real-world datasets across various domains:

  • Power: ETTm1, ETTm2, ETTh1, ETTh2
  • Economy: Exchange
  • Weather: Weather
  • Electricity: ECL
  • Transportation: Traffic

Key Insights and Ideas

  • Frequency Domain is Key for Multivariate Interactions: The paper strongly argues that the frequency domain provides a more intuitive and effective way to capture inter-variable dependencies, as variables often share similar periodic patterns in this domain.
  • Independent Processing of Frequency Components is Crucial: Treating amplitude, phase, real, and imaginary parts as separate inputs to attention sub-heads significantly improves performance compared to mixed or dependent processing. This allows the model to capture unique patterns within each frequency characteristic.
  • Real and Imaginary Parts are Most Informative: While all four frequency representations contribute, the combination of real and imaginary parts proved to be particularly effective for prediction, outperforming magnitude and phase combinations in many cases. This highlights the power of complex-valued representations.
  • Lightweight Design is Achievable Without Sacrificing Accuracy: The proposed frequency-domain approach, combined with cutting-off frequency and equivalent mapping, drastically reduces computational cost (MACs and NOP) while achieving state-of-the-art or near state-of-the-art prediction performance.
  • Equivalent Mapping is Optimal: Mapping the input sequence length to an equivalent target dimension in the frequency domain proved to be the most effective strategy, balancing information preservation and computational efficiency.

Key Results

  • State-of-the-Art Performance: The proposed model achieved the best or second-best performance on most of the evaluated multivariate time series forecasting tasks, often outperforming previous state-of-the-art methods.
  • Drastic Reduction in Computational Cost: The model requires only about 1% of the spatio-temporal cost (MACs and NOP) compared to mainstream transformer-based methods like iTransformer. For example, it uses 1.9% of MACs and 1.1% of NOP compared to iTransformer on the ECL dataset.
  • Effective Long-Term Forecasting: The model demonstrated strong performance, especially in longer prediction horizons, indicating its ability to capture complex temporal dependencies.
  • Robustness to Hyperparameters: Sensitivity analysis showed that the model's performance is relatively stable within reasonable ranges of learning rate and dropout.

Limitations and Future Work

  • Dataset-Specific Optimization: The paper notes that different datasets exhibit varying responses to cutting-off frequency, suggesting that scenario-specific optimization might be beneficial for deployment.
  • Exchange Dataset Anomaly: The Exchange dataset showed slightly reduced performance with longer input sequences, which the authors attribute to unique irregular high-frequency noise in that dataset. This suggests that handling such noise might be an area for future research.
  • Further Exploration of Frequency Domain: While the paper significantly advances frequency domain analysis, there's potential for deeper exploration of spectral data richness and more sophisticated frequency-aware modeling techniques.

Summary updated

BibTeX
@article{wang2024frequency,
  author = {Chenyue Wang and Zhouyuan Zhang and Xin Wang and Mingyang Liu and Lin Chen and Jiatian Pi},
  title = {{Frequency-Enhanced Transformer with Symmetry-Based Lightweight Multi-Representation for Multivariate Time Series Forecasting}},
  journal = {Symmetry},
  year = {2024},
  doi = {10.3390/sym16070797},
}
Figure