TL;DR
Real-world multivariate time series frequently exhibit non-stationarity that goes far beyond simple shifts in mean and variance. TwinS introduces a Transformer architecture grounded in wavelet analysis that explicitly tackles nested multi-scale periods, intermittent/missing periodicity, and cross-variable time lags (hysteresis), achieving state-of-the-art forecasting performance with up to a 25.8% MSE reduction over PatchTST.
What Problem Does This Paper Address?
Most deep learning models for multivariate time series forecasting (MTSF) assume that making a series stationary simply requires normalizing shifting statistical attributes like running mean and variance. However, empirical time-frequency analysis (using continuous Morlet wavelets) reveals three overlooked dimensions of non-stationarity:
- Nested Periodicity: Time series contain multiple overlapping cycles operating simultaneously at different time-frequency scales.
- Intermittent / Missing Periods: Periodic behaviors often switch "on" and "off" over time rather than persisting continuously. Standard dot-product attention struggles with this, often aggregating noise when a periodic pattern vanishes.
- Hysteresis Across Variables: Different channels often share periodic patterns but exhibit substantial phase lags and temporal distribution shifts. Standard channel-attention mechanisms evaluate cross-variable relationships time step by time step, which distorts dependencies when lags exist.
Methods
TwinS addresses these challenges through three tightly integrated modules:
- Wavelet Convolution Embedding: Instead of conventional linear patch projection (which can fragment local semantics and miss nested cycles), TwinS embeds raw inputs using multi-scale 1D convolutions. The convolution kernels are scaled exponentially ($2^i - 1$) and share weights centered around a base kernel—mimicking continuous wavelet decomposition to capture nested high- and low-frequency components simultaneously.
- Reversible Window Patching (RWP): To retain Transformer efficiency, the feature map undergoes reversible patching. This allows the model to alter patch scales dynamically across layers, preserve multi-scale boundaries, and apply intra-layer cyclic shifts (window rotations) to improve robustness against outliers.
- Periodic-Aware Attention (PAA): To prevent self-attention from attending to absent periods, TwinS employs a multi-head depthwise separable convolutional sub-network to generate periodic relevance scores.
- In TwinS+, these scores modulate standard Query-Key attention maps.
- In TwinS, the attention matrix is directly generated by the sub-network, cutting computational complexity to linear relative to head dimension while improving convergence.
- Channel-Temporal Mixer MLP (CT-MLP): Rather than modeling channel interactions at isolated time points, TwinS flattens and mixes the temporal and channel dimensions using a lightweight MLP. This captures joint inter-variable dependencies despite phase delays and hysteresis.
Input MTS
│
[Wavelet Convolution Embedding] ──> Multi-scale, nested period representation
│
[Reversible Window Patching] ──> Grouping into patches across layers
│
[Periodic-Aware Attention] ──> Conv sub-network detects missing/active periods
│
[Feed-Forward & CT-MLP] ──> Joint cross-channel & temporal mixing
│
Linear Projection Head ──> Future horizon prediction
Datasets
The authors evaluate TwinS across eight widely used long-term forecasting benchmarks (lookback window of 96; horizons of 96, 192, 336, and 720 steps):
- ETDataset (ETTh1, ETTh2, ETTm1, ETTm2): 2 years of electricity transformer temperatures and load metrics logged hourly ("h") or every 15 minutes ("m").
- Electricity: Hourly electricity consumption of 321 customers over 2 years.
- Weather: 21 meteorological features recorded every 10 minutes across 4 years.
- Exchange: Daily exchange rates of 8 currencies from 1990 to 2016.
- Traffic: Hourly road occupancy rates recorded by 862 highway sensors in the San Francisco Bay Area.
No new datasets were introduced.
Key Insights and Ideas
- Wavelet-inspired weight sharing matters: Ablation studies show that dynamically detecting periods (e.g., via FFT or adaptive weighting) or using unshared multi-branch kernels (like Inception) underperforms. Sharing kernel weights across scales—mirroring mother-wavelet scaling—acts as an effective regularizer while capturing nested cycles.
- Convolutional inductive biases outperform MLPs for attention routing: When predicting attention scores in PAA, replacing the convolutional sub-network with an MLP caused significant degradation. The translation invariance of 1D convolutions makes them far better suited to detecting periodic patterns and omissions.
- Joint vs. decoupled channel modeling: In datasets with strong variable lags (e.g., Weather), removing the CT-MLP led to performance drops of up to 11.2%, proving that time-agnostic channel-mixing hurts when hysteresis is present.
Key Results
- Consistent SOTA across benchmarks: TwinS / TwinS+ achieved the lowest MSE and MAE on the majority of task settings against 8 competitive baselines (including PatchTST, TimesNet, MICN, Crossformer, FEDformer, and Non-Stationary Transformer).
- Pronounced gains on non-stationary data: On datasets with severe non-stationary periodic variation (e.g., Weather and Electricity), TwinS demonstrated large performance margins, including up to a 25.8% MSE improvement over PatchTST.
- Lower complexity: By dispensing with full matrix-matrix query-key multiplications in the default TwinS variant, the attention mechanism operates faster and converges more rapidly than traditional multi-head self-attention.
Limitations and Future Work
- Scalability to very large variable counts: On high-dimensional datasets like Traffic (862 series), the CT-MLP hidden dimension was capped at a constant size to prevent parameter explosion. This limited its expressive capacity and prevented TwinS from dominating Traffic as decisively as other datasets.
- Future Directions: The authors suggest replacing the uniform CT-MLP with more scalable, graph-based inter-variable modeling approaches (e.g., SageFormer) and continuing to explore more interpretable ways to handle non-stationary periodic shifts.
Summary updated