Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting

Mohammad Amin Shabani, Amir H. Abdi, Lili Meng, Tristan Sylvain

TL;DR

Scaleformer is a model-agnostic framework that significantly improves transformer-based time series forecasting by iteratively refining predictions across multiple temporal scales. By using shared weights and a novel cross-scale normalization scheme, it achieves up to a 38.5% reduction in prediction error with minimal architectural overhead.

The Challenge of Scale in Time Series

While Transformers have become the state-of-the-art for time series forecasting, they often struggle with "scale-awareness." Real-world time series—like weather patterns or traffic flow—contain dependencies at vastly different resolutions, from minute-by-minute fluctuations to broad seasonal cycles.

Existing models like Autoformer and FEDformer attempt to address this by decomposing signals into trend and seasonal components, but they typically only operate at two fixed scales. Most Transformers learn cross-scale relationships implicitly through stacked attention layers, which lacks a strong structural prior for multi-resolution data. The core challenge is: how can we force a Transformer to explicitly reason from coarse global patterns down to fine-grained local details?

Iterative coarse-to-fine forecasting process

The Scaleformer Architecture

To solve this, the authors introduce Scaleformer, a framework designed to wrap around any existing Transformer backbone (such as Informer, Autoformer, or FEDformer). Its architecture is defined by four main pillars:

  1. Iterative Multi-scale Refinement: Instead of outputting a single forecast, the model produces a sequence of forecasts. It starts at a heavily downsampled "coarse" scale to capture macro-trends and iteratively refines the output through successive steps until it reaches the original "fine" resolution.
  2. Weight Sharing: To keep the parameter count nearly identical to the baseline models, Scaleformer uses the same Transformer module at every scale step. This forces the model to learn scale-invariant features rather than memorizing scale-specific noise.
  3. Cross-scale Normalization: This is the framework's "secret sauce." Iterative refinement often leads to "runaway error" because the distribution of the intermediate forecast shifts away from the original data. Scaleformer mitigates this by normalizing each step's input using the temporal average of both the look-back window and the previous step's forecast.
  4. Adaptive Loss: Rather than relying solely on Mean Squared Error (MSE), which is highly sensitive to outliers, the authors employ Barron’s adaptive loss. This allows the model to automatically adjust its sensitivity to outliers during training.
Scaleformer architecture showing cross-scale normalization and iterative steps

Datasets and Evaluation

The model was rigorously evaluated on several standard public benchmarks, including:

  • Electricity (ECL): Hourly consumption of 321 clients.
  • Traffic: Hourly occupancy rates of San Francisco Bay Area freeways.
  • Weather: 21 meteorological indicators recorded every 10 minutes.
  • Exchange-Rate: Daily rates of 8 countries.
  • ILI (National Illness): Weekly influenza-like illness patients.
  • Mackey-Glass (Synthetic): A newly introduced chaotic time series dataset used to test the model's ability to handle complex, non-linear dynamics.

Key Insights and Results

Scaleformer consistently outperformed vanilla Transformer baselines across all datasets, proving the value of its structural priors.

  • Scale as a Structural Prior: The iterative "coarse-to-fine" approach acts as a powerful inductive bias. By getting the "big picture" right first, the model is less likely to be distracted by high-frequency noise when making the final prediction.
  • Normalization is Non-Negotiable: The authors found that simply adding multiple scales without their specific cross-scale normalization often hurt performance. Proper normalization is essential to bridge the distribution shift between the encoder (historical data) and the decoder (evolving intermediate forecasts).
  • Significant Error Reduction: The framework achieved average MSE reductions of 5.6% over FEDformer, 13.5% over Autoformer, and a massive 38.5% over Informer.
  • Qualitative Superiority: Visualizations demonstrate that Scaleformer is much better at capturing local variance and "human-relevant" trends that baselines often smooth over or miss entirely.

Limitations and Future Work

While highly effective, the approach does come with trade-offs:

  • Computational Overhead: Although the parameter count stays the same due to weight sharing, the training and inference time roughly doubles compared to single-scale baselines because of the iterative steps. However, by using a scale factor (e.g., $s=2$), the total computational cost follows a geometric series, keeping runtime bounded.
  • Scale Factor Selection: While a scale factor of $s=2$ worked best in their tests, the optimal number of scales and the downsampling method (currently average pooling) might vary for different types of data (e.g., categorical vs. continuous).
  • Beyond Transformers: The authors briefly showed that the framework could be extended to non-transformer models like N-HiTS and FiLM, suggesting a promising path for making all time series models more scale-aware.

Summary updated

Abstract

The performance of time series forecasting has recently been greatly improved by the introduction of transformers. In this paper, we propose a general multi-scale framework that can be applied to state-of-the-art transformer-based time series forecasting models (FEDformer, Autoformer, etc.). Using iteratively refining a forecasted time series at multiple scales with shared weights, architecture adaptations and a specially-designed normalization scheme, we are able to achieve significant performance improvements with minimal additional computational overhead. Via detailed ablation studies, we demonstrate the effectiveness of our proposed architectural and methodological innovations. Furthermore, our experiments on various public datasets demonstrate that the proposed method outperforms the corresponding baselines. Depending on the choice of transformer architecture, our mutli-scale framework results in mean squared error reductions ranging from 5.5% to 38.5%. Our code is publicly available in https://github.com/BorealisAI/scaleformer.
BibTeX
@inproceedings{shabani2023scaleformer,
  author = {Mohammad Amin Shabani and Amir H. Abdi and Lili Meng and Tristan Sylvain},
  title = {{Scaleformer: Iterative Multi-scale Refining Transformers for Time Series Forecasting}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=sCrnllCtjoE},
}
Figure