AutoTimes: Autoregressive Time Series Forecasters via Large Language Models

Yong Liu, Guo Qin, Xiangdong Huang, Jianmin Wang, Mingsheng Long

Introduction

The success of foundation models in natural language processing (NLP) and computer vision has prompted researchers to explore their applicability to time series forecasting. However, time series data presents unique challenges, including limited large-scale datasets and architectural mismatches. Recent efforts to "reprogram" Large Language Models (LLMs) for time series (LLM4TS) often treat these models as black-box encoders, flattening lookback windows into single prompts and generating forecasts in a non-autoregressive manner.

This approach fails to leverage the core strength of decoder-only LLMs: their pre-trained capability for next-token prediction. Furthermore, many existing methods require training separate models for different forecast horizons. To address these inefficiencies, we introduce AutoTimes, a framework that repurposes frozen LLMs into flexible, high-performance autoregressive forecasters.

Methodology

AutoTimes aligns the time series modality with the language modality while preserving the LLM's native autoregressive structure. The framework consists of three primary components: modality alignment via segments, textual timestamp embeddings, and autoregressive generation.

Overview of the AutoTimes framework, comparing its autoregressive approach and time-series-based prompting mechanism with previous methods.

Modality Alignment and Generation

Instead of processing raw numerical values, AutoTimes partitions a univariate time series into non-overlapping segments, or "patches." A lightweight Multi-Layer Perceptron (MLP) projects these segments into the LLM’s embedding space, effectively treating them as tokens.

Crucially, AutoTimes maintains the LLM's autoregressive nature. By keeping the LLM frozen and using it to predict the next time series segment iteratively, the model can generate forecasts of arbitrary lengths. This "one-for-all" capability allows a single model to handle various lookback and forecast horizons without retraining.

Textual Timestamps and In-Context Forecasting

To incorporate temporal information, AutoTimes utilizes the LLM’s inherent understanding of time. Timestamps (e.g., "2016/07/01 00:00:00") are converted into text and passed through the frozen LLM. The resulting vectors serve as position embeddings, helping the model capture periodicity and align multiple variables.

Additionally, AutoTimes introduces In-Context Forecasting. By prepending relevant historical data from similar series to the lookback window, the model can adapt to specific dataset patterns on the fly, mimicking the "few-shot" prompting capabilities of LLMs in NLP.

Experimental Setup

The performance of AutoTimes was evaluated across a diverse set of benchmarks:

  • Long-term Forecasting: ETTh1, ECL (Electricity), Traffic, Weather, and Solar-Energy.
  • Short-term Forecasting: The M4 competition dataset.
  • Zero-shot & In-context Learning: M3 and M4 datasets were used to assess cross-domain transferability.
  • Additional Benchmarks: Australian Electricity, Bdg-2 Panther, and Oikolab Weather.

Key Results and Insights

AutoTimes achieves state-of-the-art (SOTA) performance while being significantly more efficient than previous LLM-based methods.

Performance and Efficiency

AutoTimes consistently outperformed advanced forecasters such as TimeLLM, PatchTST, and iTransformer. Notably, it achieved a 5x speedup in training and inference compared to TimeLLM. This efficiency stems from avoiding complex text-based prompts that lead to excessive context lengths. Furthermore, by freezing the LLM backbone and only training the alignment layers, AutoTimes utilizes only 0.1% of the total parameters, enabling the use of large models like LLaMA-7B on standard hardware.

Scaling and Zero-Shot Transfer

The study revealed a clear scaling trend: as the base LLM size increases (e.g., from GPT-2 124M to LLaMA 7B), forecasting accuracy generally improves. In zero-shot scenarios (M4 → M3), AutoTimes reduced the Symmetric Mean Absolute Percentage Error (SMAPE) by 15% compared to deep learning baselines like DLinear. In-context forecasting further reduced errors by up to 13.3% in zero-shot tasks.

Conclusion and Future Work

AutoTimes demonstrates that preserving the autoregressive nature of LLMs is essential for effective time series forecasting. By treating time series segments as tokens and utilizing textual timestamps, the framework creates a scalable and efficient "one-for-all" forecaster.

Future work includes extending the model to provide probabilistic forecasts rather than just point estimates. Additionally, integrating multimodal data—such as news or logs—and exploring parameter-efficient fine-tuning techniques like LoRA could further enhance performance.

Summary updated

BibTeX
@inproceedings{liu2024autotimes,
  author = {Yong Liu and Guo Qin and Xiangdong Huang and Jianmin Wang and Mingsheng Long},
  title = {{AutoTimes: Autoregressive Time Series Forecasters via Large Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=FOvZztnp1H},
}
Figure