TL;DR
STeP-TS is a compact, one-layer convolutional framework for multivariate time series forecasting that matches the performance of heavy-duty Transformers and RNNs with a fraction of the computational cost. By using a shared bank of multi-scale filters combined with variable-specific weights and a novel amplitude-adaptive normalization, it achieves state-of-the-art accuracy while requiring up to 27x fewer FLOPs and 10x fewer parameters than leading models.
What Problem Does This Paper Address?
In the world of multivariate time series forecasting (like predicting weather or power grid loads), models need to capture two things: shared patterns across different variables (e.g., a daily temperature cycle affecting multiple sensors) and the unique dynamics of each individual variable.
Recent trends have favored increasingly massive architectures, such as PatchTST (Transformers) or ModernTCN (large-kernel convolutions). However, these models come with high memory and computational costs. The authors of STeP-TS challenge this "bigger is better" mentality, asking how much model capacity is actually necessary for accurate forecasting, especially in resource-constrained edge computing environments.
Methods
The authors propose a surprisingly simple yet effective architecture:
- Shared Multi-Scale Filters: Instead of learning unique filters for every variable, the model learns a single bank of 1D convolutional filters with varying kernel sizes (multi-scale). These filters are shared across all variables to capture common temporal patterns efficiently.
- Variable-Specific Weighting: To handle the unique dynamics of each variable, the model learns lightweight scalar weights that determine how much each shared filter contributes to a specific variable's representation.
- Efficient Large Kernels: To capture long-range dependencies without the massive cost of large kernels, STeP-TS uses strided convolutions followed by linear interpolation to restore the original temporal resolution.
- Structured Pruning: The model includes a built-in mechanism to identify redundant filters during training. By analyzing the learned weights and the "energy" (variance) of filter outputs, the model can prune entire convolutional channels, reducing complexity with almost no loss in accuracy.
- adRevIN (Amplitude-Adaptive Normalization): Standard normalization techniques often struggle when the signal amplitude changes suddenly (e.g., a sudden spike in power usage). STeP-TS introduces a bounded, input-conditioned gain—predicted by a tiny MLP—to adjust the normalization scale dynamically.
Datasets
The model was evaluated on seven standard long-term forecasting benchmarks:
- Electricity: Power usage data.
- Traffic: Road occupancy rates.
- Weather: Meteorological indicators.
- ETT (ETTh1, ETTh2, ETTm1, ETTm2): Electricity Transformer Temperature datasets at different sampling frequencies.
Key Insights and Ideas
- Simplicity Wins: A single, well-designed convolutional layer can be just as effective as deep, multi-layer Transformers for many forecasting tasks.
- Shared Structure is Sufficient: The assumption that different variables in a multivariate series share underlying temporal "motifs" holds true, allowing for massive parameter savings through shared filter banks.
- Pruning as a Design Tool: Because the architecture is shallow and transparent, the contribution of each filter is directly observable. This makes "structured pruning"—removing entire functional blocks—far more effective and interpretable than in deep, black-box models.
- Normalization is a Bottleneck: The authors found that standard Reversible Instance Normalization (RevIN) was a limiting factor. Their
adRevINmodule shows that allowing the model to learn how to scale its predictions based on the input's current "regime" significantly boosts performance.
Key Results
STeP-TS demonstrates an impressive "accuracy vs. cost" profile. On the Weather dataset, it matches the accuracy of the strongest baselines while being significantly leaner:
- 3.1x fewer parameters and 3.0x fewer FLOPs than xLSTMMixer.
- 10.2x to 27.3x fewer FLOPs than ModernTCN.
- Significantly lower latency: As shown in Figure 1, STeP-TS sits in the "sweet spot" of low error and low per-sample latency.
Furthermore, the pruning mechanism proved highly effective. In experiments, the model could prune 3 out of 8 channels, resulting in a 1.2x speedup with a negligible MSE degradation of only 0.03%.
Limitations and Future Work
While STeP-TS is highly efficient, the authors note a few areas for growth:
- Pruning Gains: While FLOPs and latency drop significantly with pruning, the total parameter count reduction is relatively small because the model is already very compact.
- Temporal Structures: The current work focuses on 1D convolutions. Future iterations could explore if other lightweight temporal structures (like simple state-space models) could be integrated into this shared-filter framework.
- Interpretability: The authors suggest that because each filter corresponds to a specific temporal scale, the model could be used to "diagnose" which frequencies are most important for specific variables, though this was not fully explored in the current paper.
Summary updated