Scaling-laws for Large Time-series Models

Thomas D. P. Edwards, James Alvey, Justin Alsing, Nam H. Nguyen, Benjamin D. Wandelt

Abstract

Just as Large Language Models (LLMs) like GPT-4 show predictable performance gains as they scale, this research demonstrates that Large Time-series Models (LTMs) follow the same power-law scaling behavior. By training decoder-only transformers on a massive 8-billion-point dataset, the study establishes that forecasting performance improves predictably across five orders of magnitude of model size, compute, and data.

Introduction

Time-series forecasting is the backbone of critical infrastructure, from supply chain management to climate science. Traditionally, the field has relied on "bespoke" models—either statistical (like ARIMA) or deep learning models trained on single, narrow datasets—which often fail to generalize across domains.

The success of LLMs has sparked interest in "Foundation Models" for time-series: general-purpose models pre-trained on diverse data for zero-shot forecasting. However, building these is resource-intensive. This paper provides the first empirical proof that scaling parameters or data yields predictable returns on investment, justifying the development of massive time-series models.

Methodology

The authors adopted a "GPT-style" approach to time-series modeling:

  • Architecture: A simple decoder-only transformer with self-attention was used to test if raw transformer architectures scale effectively without complex "patching."
  • Distribution Head: Instead of point forecasts, the model outputs parameters for a Student’s-t distribution (mean, variance, and degrees of freedom) to account for the "heavy tails" and extreme outliers typical of real-world data.
  • Loss Function: Negative log-likelihood was used to train the model to learn the entire probability distribution of the next time step.
  • Scaling Variables: The study systematically varied model size ($N_p$ from 1,000 to 100 million parameters), compute ($C$), and dataset size ($D$).

Datasets

The researchers assembled a massive corpus of 8 billion data points across 30 million individual time series from 38 distinct sources, including the Monash Archive (weather, traffic), NOAA/ERA5 climate data, BuildingsBench energy demand, and Yahoo Finance. Diversity was prioritized, ensuring no single source dominated more than 15% of the total dataset.

Key Results and Insights

The most significant finding is that time-series models obey the same $L(A) \propto A^{-B}$ power law as LLMs. Whether measuring performance via Mean Squared Error (MSE) or probabilistic scores (CRPS), the relationship appears as a straight line on a log-log plot.

Scaling laws for Large Time-series Models across parameters, compute, and data size.
  • Architecture is Secondary to Scale: Specific design choices (e.g., width vs. depth) have minimal effect compared to the primary driver of total scale.
  • Diversity is Essential: Scaling laws only emerged when the relative proportions of different data sources were kept constant; volume alone is insufficient without diversity.
  • In-Sequence to Forecasting: There is a clear link between "next-token" prediction (in-sequence loss) and long-term forecasting accuracy. As the model improves at predicting the immediate next point, its ability to roll out coherent long-term forecasts improves dramatically.

Limitations and Future Work

While these results are foundational, the study was limited to univariate time-series and a context window of 256 data points. Future work is needed to determine if these laws hold for multivariate problems and longer context lengths. Additionally, while 100 million parameters is large for time-series, it is small compared to LLMs; verifying these laws in the billion-parameter range remains a vital next step for the community.

Summary updated

BibTeX
@inproceedings{edwards2024scaling,
  author = {Thomas D. P. Edwards and James Alvey and Justin Alsing and Nam H. Nguyen and Benjamin D. Wandelt},
  title = {{Scaling-laws for Large Time-series Models}},
  booktitle = {TSALM @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=tUHJvUy1Nb},
}
Figure