TwinS: Revisiting Non-Stationarity in Multivariate Time Series Forecasting

Jiaxi Hu, Qingsong Wen, Sijie Ruan, Li Liu, Yuxuan LiangSubmission to ICLR 2024 · Rejected

TL;DR

Real-world multivariate time series frequently exhibit non-stationarity that goes far beyond simple shifts in mean and variance. TwinS introduces a Transformer architecture grounded in wavelet analysis that explicitly tackles nested multi-scale periods, intermittent/missing periodicity, and cross-variable time lags (hysteresis), achieving state-of-the-art forecasting performance with up to a 25.8% MSE reduction over PatchTST.

What Problem Does This Paper Address?

Most deep learning models for multivariate time series forecasting (MTSF) assume that making a series stationary simply requires normalizing shifting statistical attributes like running mean and variance. However, empirical time-frequency analysis (using continuous Morlet wavelets) reveals three overlooked dimensions of non-stationarity:

  1. Nested Periodicity: Time series contain multiple overlapping cycles operating simultaneously at different time-frequency scales.
  2. Intermittent / Missing Periods: Periodic behaviors often switch "on" and "off" over time rather than persisting continuously. Standard dot-product attention struggles with this, often aggregating noise when a periodic pattern vanishes.
  3. Hysteresis Across Variables: Different channels often share periodic patterns but exhibit substantial phase lags and temporal distribution shifts. Standard channel-attention mechanisms evaluate cross-variable relationships time step by time step, which distorts dependencies when lags exist.

Methods

TwinS addresses these challenges through three tightly integrated modules:

  • Wavelet Convolution Embedding: Instead of conventional linear patch projection (which can fragment local semantics and miss nested cycles), TwinS embeds raw inputs using multi-scale 1D convolutions. The convolution kernels are scaled exponentially ($2^i - 1$) and share weights centered around a base kernel—mimicking continuous wavelet decomposition to capture nested high- and low-frequency components simultaneously.
  • Reversible Window Patching (RWP): To retain Transformer efficiency, the feature map undergoes reversible patching. This allows the model to alter patch scales dynamically across layers, preserve multi-scale boundaries, and apply intra-layer cyclic shifts (window rotations) to improve robustness against outliers.
  • Periodic-Aware Attention (PAA): To prevent self-attention from attending to absent periods, TwinS employs a multi-head depthwise separable convolutional sub-network to generate periodic relevance scores.
  • In TwinS+, these scores modulate standard Query-Key attention maps.
  • In TwinS, the attention matrix is directly generated by the sub-network, cutting computational complexity to linear relative to head dimension while improving convergence.
  • Channel-Temporal Mixer MLP (CT-MLP): Rather than modeling channel interactions at isolated time points, TwinS flattens and mixes the temporal and channel dimensions using a lightweight MLP. This captures joint inter-variable dependencies despite phase delays and hysteresis.
Input MTS
   │
[Wavelet Convolution Embedding] ──> Multi-scale, nested period representation
   │
[Reversible Window Patching]    ──> Grouping into patches across layers
   │
[Periodic-Aware Attention]      ──> Conv sub-network detects missing/active periods
   │
[Feed-Forward & CT-MLP]         ──> Joint cross-channel & temporal mixing
   │
Linear Projection Head          ──> Future horizon prediction

Datasets

The authors evaluate TwinS across eight widely used long-term forecasting benchmarks (lookback window of 96; horizons of 96, 192, 336, and 720 steps):

  • ETDataset (ETTh1, ETTh2, ETTm1, ETTm2): 2 years of electricity transformer temperatures and load metrics logged hourly ("h") or every 15 minutes ("m").
  • Electricity: Hourly electricity consumption of 321 customers over 2 years.
  • Weather: 21 meteorological features recorded every 10 minutes across 4 years.
  • Exchange: Daily exchange rates of 8 currencies from 1990 to 2016.
  • Traffic: Hourly road occupancy rates recorded by 862 highway sensors in the San Francisco Bay Area.

No new datasets were introduced.

Key Insights and Ideas

  • Wavelet-inspired weight sharing matters: Ablation studies show that dynamically detecting periods (e.g., via FFT or adaptive weighting) or using unshared multi-branch kernels (like Inception) underperforms. Sharing kernel weights across scales—mirroring mother-wavelet scaling—acts as an effective regularizer while capturing nested cycles.
  • Convolutional inductive biases outperform MLPs for attention routing: When predicting attention scores in PAA, replacing the convolutional sub-network with an MLP caused significant degradation. The translation invariance of 1D convolutions makes them far better suited to detecting periodic patterns and omissions.
  • Joint vs. decoupled channel modeling: In datasets with strong variable lags (e.g., Weather), removing the CT-MLP led to performance drops of up to 11.2%, proving that time-agnostic channel-mixing hurts when hysteresis is present.

Key Results

  • Consistent SOTA across benchmarks: TwinS / TwinS+ achieved the lowest MSE and MAE on the majority of task settings against 8 competitive baselines (including PatchTST, TimesNet, MICN, Crossformer, FEDformer, and Non-Stationary Transformer).
  • Pronounced gains on non-stationary data: On datasets with severe non-stationary periodic variation (e.g., Weather and Electricity), TwinS demonstrated large performance margins, including up to a 25.8% MSE improvement over PatchTST.
  • Lower complexity: By dispensing with full matrix-matrix query-key multiplications in the default TwinS variant, the attention mechanism operates faster and converges more rapidly than traditional multi-head self-attention.

Limitations and Future Work

  • Scalability to very large variable counts: On high-dimensional datasets like Traffic (862 series), the CT-MLP hidden dimension was capped at a constant size to prevent parameter explosion. This limited its expressive capacity and prevented TwinS from dominating Traffic as decisively as other datasets.
  • Future Directions: The authors suggest replacing the uniform CT-MLP with more scalable, graph-based inter-variable modeling approaches (e.g., SageFormer) and continuing to explore more interpretable ways to handle non-stationary periodic shifts.

Summary updated

Abstract

Recently, multivariate time series forecasting tasks have garnered increasing attention due to their significant practical applications, leading to the emergence of various deep forecasting models. However, real-world time series exhibit pronounced non-stationary distribution characteristics. These characteristics are not solely limited to time-varying statistical properties highlighted by non-stationary Transformer but also encompass three key aspects: nested periodicity, absence of periodic distributions, and hysteresis among time variables. In this paper, we begin by validating this theory through wavelet analysis and propose the Transformer-based TwinS model, which consists of three modules to address the non-stationary periodic distributions: Wavelet Convolution, Period-Aware Attention, and Channel-Temporal Mixed MLP. Specifically, The Wavelet Convolution models nested periods by scaling the convolution kernel size like wavelet transform. The Period Aware Attention guides attention computation by generating period relevance scores through a convolutional sub-network. The Channel-Temporal Mixed MLP captures the overall relationships between time series through channel-time mixing learning. TwinS achieves SOTA performance on eight datasets compared to mainstream TS models, with a maximum improvement in MSE of 25.8% over PatchTST. We have open-sourced our code to facilitate reproducibility for future research: https://anonymous.4open.science/r/TwinS-BBA3/.
BibTeX
@misc{hu2024twins,
  author = {Jiaxi Hu and Qingsong Wen and Sijie Ruan and Li Liu and Yuxuan Liang},
  title = {{TwinS: Revisiting Non-Stationarity in Multivariate Time Series Forecasting}},
  year = {2024},
  eprint = {2406.03710},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.03710},
}

Loading related content...

Figure