TwinS: Revisiting Non-Stationarity in Multivariate Time Series Forecasting

Jiaxi Hu, Qingsong Wen, Sijie Ruan, Li Liu, Yuxuan Liang

Abstract

Multivariate time series forecasting has become increasingly vital across various domains, yet real-world data often exhibits pronounced non-stationary characteristics that challenge existing models. Beyond simple time-varying statistical properties, these characteristics include nested periodicity, the absence of periodic distributions in certain intervals, and hysteresis among time variables. This paper introduces TwinS, a Transformer-based framework designed to address these frequency-domain shifts. TwinS integrates three key modules: Wavelet Convolution (WC) for modeling nested periods, Period-Aware Attention (PAA) for intermittent periodicities, and a Channel-Temporal Mixed MLP (CT-MLP) to capture inter-variable dependencies. TwinS achieves state-of-the-art performance, notably improving Mean Squared Error (MSE) by up to 25.8% over PatchTST.

The Challenge of Non-Stationarity

Forecasting real-world multivariate time series (e.g., weather patterns, traffic flow, or electricity usage) is notoriously difficult because the underlying data is non-stationary. While many existing models attempt to mitigate this by normalizing simple statistics like mean and variance, they often ignore more complex frequency-domain shifts. The authors identify three critical gaps in current methods:

  1. Nested Periodicity: Real data often contains multiple overlapping cycles (e.g., daily and weekly patterns) occurring simultaneously.
  2. Missing Periodic States: Certain cycles might only appear during specific intervals and vanish elsewhere, leading to "gaps" in periodic information.
  3. Hysteresis: There is often a time lag or "memory effect" in how different variables interact with one another.

Wavelet analysis demonstrates that periodic patterns in real-world data are often segmented rather than constant. As shown in Figure 1, a specific frequency might be strong during one interval but completely absent in another.

Wavelet analysis of time series data revealing how periodic energy (strength) shifts across different time-frequency scales.

Methodology: The TwinS Framework

TwinS is built on three innovative modules designed to explicitly model these non-stationary traits:

  • Wavelet Convolution (WC): Inspired by wavelet transforms, this module uses convolutional kernels of varying sizes (scaled by powers of 2) that share the same weights. This allows the model to effectively decouple and capture nested periodic information at different scales right from the initial embedding stage.
  • Period-Aware Attention (PAA): Standard Transformers struggle when periodic patterns are absent or shifted because their attention mechanisms aggregate data uniformly. PAA uses a lightweight convolutional sub-network to generate "period relevance scores," guiding the model to focus only on time patches that share meaningful periodic signatures.
  • Channel-Temporal Mixed MLP (CT-MLP): To address hysteresis, TwinS moves away from modeling time and channels in isolation. This module treats the multivariate series as a holistic entity, using a Multi-Layer Perceptron to learn the complex, delayed interdependencies between different variables.
The overall architecture of TwinS, highlighting the Wavelet Conv, Encoder-Layer, and Linear Decoder modules.

Experimental Results

The model was evaluated on eight widely-used public benchmarks, including Weather, Traffic, Electricity, and the ETT datasets. TwinS consistently outperformed eight mainstream forecasting models, including Autoformer, FEDformer, and PatchTST.

  • Accuracy Boost: TwinS achieved a maximum improvement in MSE of 25.8% over PatchTST on the Weather dataset.
  • Efficiency: The PAA module is mathematically simplified to reduce time and space complexity compared to standard Multi-Head Self-Attention, making it more efficient for long sequences.
  • Robustness: Visualization of attention mechanisms confirms that TwinS focuses on distinct, relevant periodic patches, whereas older models distribute weights more uniformly and less effectively across non-stationary distributions.

Conclusion and Future Work

TwinS provides a robust solution for multivariate time series forecasting by revisiting the fundamental nature of non-stationarity. By hybridizing the strengths of CNNs (for periodic filtering) and Transformers (for long-range dependencies), it offers a more nuanced understanding of frequency-domain shifts. Future work will explore replacing the CT-MLP with more advanced modules, such as graph-based approaches, to better handle datasets with an extremely high number of variables where simple MLPs may reach their expressive limits.

Summary updated

Abstract

Recently, multivariate time series forecasting tasks have garnered increasing attention due to their significant practical applications, leading to the emergence of various deep forecasting models. However, real-world time series exhibit pronounced non-stationary distribution characteristics. These characteristics are not solely limited to time-varying statistical properties highlighted by non-stationary Transformer but also encompass three key aspects: nested periodicity, absence of periodic distributions, and hysteresis among time variables. In this paper, we begin by validating this theory through wavelet analysis and propose the Transformer-based TwinS model, which consists of three modules to address the non-stationary periodic distributions: Wavelet Convolution, Period-Aware Attention, and Channel-Temporal Mixed MLP. Specifically, The Wavelet Convolution models nested periods by scaling the convolution kernel size like wavelet transform. The Period-Aware Attention guides attention computation by generating period relevance scores through a convolutional sub-network. The Channel-Temporal Mixed MLP captures the overall relationships between time series through channel-time mixing learning. TwinS achieves SOTA performance compared to mainstream TS models, with a maximum improvement in MSE of 25.8\% over PatchTST.
BibTeX
@misc{hu2024twins,
  author = {Jiaxi Hu and Qingsong Wen and Sijie Ruan and Li Liu and Yuxuan Liang},
  title = {{TwinS: Revisiting Non-Stationarity in Multivariate Time Series Forecasting}},
  year = {2024},
  eprint = {2406.03710},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.03710},
}
Figure