TL;DR
Scaleformer is a model-agnostic framework that significantly improves transformer-based time series forecasting by iteratively refining predictions across multiple temporal scales. By using shared weights and a novel cross-scale normalization scheme, it achieves up to a 38.5% reduction in prediction error with minimal architectural overhead.
The Challenge of Scale in Time Series
While Transformers have become the state-of-the-art for time series forecasting, they often struggle with "scale-awareness." Real-world time series—like weather patterns or traffic flow—contain dependencies at vastly different resolutions, from minute-by-minute fluctuations to broad seasonal cycles.
Existing models like Autoformer and FEDformer attempt to address this by decomposing signals into trend and seasonal components, but they typically only operate at two fixed scales. Most Transformers learn cross-scale relationships implicitly through stacked attention layers, which lacks a strong structural prior for multi-resolution data. The core challenge is: how can we force a Transformer to explicitly reason from coarse global patterns down to fine-grained local details?
The Scaleformer Architecture
To solve this, the authors introduce Scaleformer, a framework designed to wrap around any existing Transformer backbone (such as Informer, Autoformer, or FEDformer). Its architecture is defined by four main pillars:
- Iterative Multi-scale Refinement: Instead of outputting a single forecast, the model produces a sequence of forecasts. It starts at a heavily downsampled "coarse" scale to capture macro-trends and iteratively refines the output through successive steps until it reaches the original "fine" resolution.
- Weight Sharing: To keep the parameter count nearly identical to the baseline models, Scaleformer uses the same Transformer module at every scale step. This forces the model to learn scale-invariant features rather than memorizing scale-specific noise.
- Cross-scale Normalization: This is the framework's "secret sauce." Iterative refinement often leads to "runaway error" because the distribution of the intermediate forecast shifts away from the original data. Scaleformer mitigates this by normalizing each step's input using the temporal average of both the look-back window and the previous step's forecast.
- Adaptive Loss: Rather than relying solely on Mean Squared Error (MSE), which is highly sensitive to outliers, the authors employ Barron’s adaptive loss. This allows the model to automatically adjust its sensitivity to outliers during training.
Datasets and Evaluation
The model was rigorously evaluated on several standard public benchmarks, including:
- Electricity (ECL): Hourly consumption of 321 clients.
- Traffic: Hourly occupancy rates of San Francisco Bay Area freeways.
- Weather: 21 meteorological indicators recorded every 10 minutes.
- Exchange-Rate: Daily rates of 8 countries.
- ILI (National Illness): Weekly influenza-like illness patients.
- Mackey-Glass (Synthetic): A newly introduced chaotic time series dataset used to test the model's ability to handle complex, non-linear dynamics.
Key Insights and Results
Scaleformer consistently outperformed vanilla Transformer baselines across all datasets, proving the value of its structural priors.
- Scale as a Structural Prior: The iterative "coarse-to-fine" approach acts as a powerful inductive bias. By getting the "big picture" right first, the model is less likely to be distracted by high-frequency noise when making the final prediction.
- Normalization is Non-Negotiable: The authors found that simply adding multiple scales without their specific cross-scale normalization often hurt performance. Proper normalization is essential to bridge the distribution shift between the encoder (historical data) and the decoder (evolving intermediate forecasts).
- Significant Error Reduction: The framework achieved average MSE reductions of 5.6% over FEDformer, 13.5% over Autoformer, and a massive 38.5% over Informer.
- Qualitative Superiority: Visualizations demonstrate that Scaleformer is much better at capturing local variance and "human-relevant" trends that baselines often smooth over or miss entirely.
Limitations and Future Work
While highly effective, the approach does come with trade-offs:
- Computational Overhead: Although the parameter count stays the same due to weight sharing, the training and inference time roughly doubles compared to single-scale baselines because of the iterative steps. However, by using a scale factor (e.g., $s=2$), the total computational cost follows a geometric series, keeping runtime bounded.
- Scale Factor Selection: While a scale factor of $s=2$ worked best in their tests, the optimal number of scales and the downsampling method (currently average pooling) might vary for different types of data (e.g., categorical vs. continuous).
- Beyond Transformers: The authors briefly showed that the framework could be extended to non-transformer models like N-HiTS and FiLM, suggesting a promising path for making all time series models more scale-aware.
Summary updated