Chronos: Learning the Language of Time Series

Abdul Fatir Ansari, Lorenzo Stella, Ali Caner Türkmen, Xiyuan Zhang, Pedro Mercado, Huibin Shen, Oleksandr Shchur, Syama Sundar Rangapuram, Sebastian Pineda-Arango, Shubham Kapoor, Jasper Zschiegner, Danielle C. Maddix, Michael W. Mahoney, Kari Torkkola, Andrew Gordon Wilson, Michael Bohlke-Schneider, Yuyang Wang

Introduction

Time series forecasting has traditionally relied on either local statistical models, such as ARIMA, or global deep learning models like DeepAR that are tailored to specific datasets. However, the field has lacked a true foundation model—a single, pretrained architecture capable of accurate zero-shot forecasting across diverse, unseen domains. The primary obstacle is the continuous and unbounded nature of time series data, which contrasts sharply with the discrete, finite vocabularies used in natural language processing.

Chronos addresses this gap by treating time series forecasting as a language modeling problem. By tokenizing numerical values into a discrete vocabulary and leveraging standard Transformer architectures, Chronos achieves state-of-the-art zero-shot performance without requiring domain-specific modifications.

Methodology

The Chronos framework adopts a minimalist approach, utilizing existing language model architectures with specific preprocessing steps to bridge the gap between continuous values and discrete tokens.

The Chronos pipeline: Tokenization, Training, and Inference.

Scaling and Quantization

To handle varying magnitudes, each time series is normalized via mean scaling, dividing values by the mean absolute value of the historical data. These scaled values are then quantized into a fixed vocabulary of 4096 tokens. This transformation converts a sequence of real numbers into a sequence of "words" that a standard Transformer can process.

Architecture and Training

Chronos employs the T5 (Text-to-Text Transfer Transformer) architecture, with models ranging from 20M to 710M parameters. Unlike traditional regression models that predict a single point, Chronos is trained using categorical cross-entropy loss to predict a probability distribution over the quantized bins. This approach naturally facilitates probabilistic forecasting, allowing the model to represent complex, multi-modal distributions.

Data Augmentation and Synthetic Generation

A key component of Chronos's success is its training corpus, which combines 28 public datasets with a massive volume of synthetic data. Two primary augmentation techniques are used:

  1. TSMixup: A method that mixes different time series to create new training samples.
  2. KernelSynth: This technique uses Gaussian Processes with randomly composed kernels (e.g., linear, periodic, RBF) to generate millions of diverse synthetic time series.
Synthetic samples generated via KernelSynth, illustrating the diversity of patterns used for pretraining.

The authors found that diversity in training patterns is more critical than the realism of the data. Models trained purely on synthetic KernelSynth data often outperformed established baselines trained on real-world datasets.

Key Results and Insights

Chronos was evaluated on two benchmarks: an in-domain set (Benchmark I) and a zero-shot set of 27 unseen datasets (Benchmark II).

  • Zero-Shot Excellence: Chronos significantly outperformed classical statistical models and other recent pretrained models like Lag-Llama and Moirai on unseen data.
  • Competitive Performance: The largest Chronos models performed on par with task-specific deep learning models like PatchTST, despite having no prior exposure to the target datasets.
  • Fine-tuning Benefits: While zero-shot performance is strong, lightweight fine-tuning of smaller models (e.g., Chronos-Small) can further enhance accuracy, often surpassing larger zero-shot models.
  • General Pattern Recognition: The success of Chronos suggests that standard Transformer blocks are sufficient for learning the "grammar" of sequential data without needing time-series-specific features like seasonal encodings or lags.

Limitations and Future Work

Despite its strengths, Chronos currently focuses on univariate forecasting and does not incorporate exogenous variables or multivariate dependencies. The quantization process also involves trade-offs between vocabulary size and precision, particularly for data with extreme outliers. Future research may explore scaling to larger models, integrating covariates through adapter layers, and extending the framework to tasks such as anomaly detection and classification.

Summary updated

Abstract

We introduce Chronos, a simple yet effective framework for pretrained probabilistic time series models. Chronos tokenizes time series values using scaling and quantization into a fixed vocabulary and trains existing transformer-based language model architectures on these tokenized time series via the cross-entropy loss. We pretrained Chronos models based on the T5 family (ranging from 20M to 710M parameters) on a large collection of publicly available datasets, complemented by a synthetic dataset that we generated via Gaussian processes to improve generalization. In a comprehensive benchmark consisting of 42 datasets, and comprising both classical local models and deep learning methods, we show that Chronos models: (a) significantly outperform other methods on datasets that were part of the training corpus; and (b) have comparable and occasionally superior zero-shot performance on new datasets, relative to methods that were trained specifically on them. Our results demonstrate that Chronos models can leverage time series data from diverse domains to improve zero-shot accuracy on unseen forecasting tasks, positioning pretrained models as a viable tool to greatly simplify forecasting pipelines.
BibTeX
@article{ansari2024chronos,
  author = {Abdul Fatir Ansari and Lorenzo Stella and Ali Caner Türkmen and Xiyuan Zhang and Pedro Mercado and Huibin Shen and Oleksandr Shchur and Syama Sundar Rangapuram and Sebastian Pineda-Arango and Shubham Kapoor and Jasper Zschiegner and Danielle C. Maddix and Michael W. Mahoney and Kari Torkkola and Andrew Gordon Wilson and Michael Bohlke-Schneider and Yuyang Wang},
  title = {{Chronos: Learning the Language of Time Series}},
  journal = {Accepted by TMLR},
  year = {2024},
  url = {https://openreview.net/forum?id=gerNCVqqtR},
}
Figure