Diffusion Auto-regressive Transformer for Effective Self-supervised Time Series Forecasting

Daoyu Wang, Mingyue Cheng, Zhiding Liu, Qi Liu, Enhong ChenSubmission to ICLR 2025 · Rejected

TL;DR

TimeDART introduces a generative self-supervised pre-training framework that pairs an auto-regressive Transformer encoder with a diffusion-based denoising patch decoder for time series forecasting. By using causal modeling to capture global inter-patch trends and diffusion to recover fine-grained intra-patch dynamics, the model substantially narrows the gap between self-supervised pre-training and downstream forecasting. Across extensive benchmarks, TimeDART achieves state-of-the-art transfer performance compared to existing contrastive, masked autoencoder, and supervised baselines.

What Problem Does This Paper Address?

Self-supervised learning (SSL) in time series usually relies on one of two strategies:

  1. Masked Autoencoders (MAE): Reconstruct masked values, but create a distribution gap between pre-training (corrupted data) and downstream forecasting (full clean sequences).
  2. Contrastive Learning: Learn discriminative features by comparing similar/dissimilar windows, which is ill-suited for time series where defining pair similarity is ambiguous and temporal generative dynamics matter most.

Moreover, standard Mean Squared Error (MSE) pre-training encourages overly smooth predictions that miss multimodal, localized noise. Time series models need to jointly capture long-range temporal trends (global context) and local high-frequency variations (intra-patch details) without degrading downstream transferability.

Methods

TimeDART breaks the look-back window into non-overlapping patches (setting stride $S = \text{patch length } P$) under a channel-independent setting. The architecture couples two complementary generative mechanisms:

  • Causal Transformer Encoder (Inter-Patch Modeling): Processes clean patch embeddings using a causal mask and a start-of-sequence (SOS) token. This forces the encoder to model global sequence dependencies auto-regressively—aligning pre-training directly with the "past predicts future" forecasting paradigm.
  • Diffusion Denoising Decoder (Intra-Patch Modeling): Adds forward Gaussian noise to each patch independently across time steps using a smooth cosine noise schedule. A cross-attention decoder then takes the noisy patches as queries and the causal encoder states as keys and values. A diagonal mask ensures the $j$-th noisy patch only attends to encoder representations aggregated up to time step $j-1$.
  • Generative Objective: Pre-training minimizes the diffusion Evidence Lower Bound (ELBO), optimizing the model to denoise corrupted patches conditioned on past context.
  • Downstream Fine-Tuning: The diffusion decoder is removed, and a simple linear projection head is attached to the frozen or fine-tuned Transformer encoder to produce multi-step forecasts via one-step generation.
Input Patches [x_1, ..., x_N] 
  ├──> Causal Transformer Encoder ──> Contextual States f(z_{1:j-1}) 
  │                                           │ (Keys & Values)
  └──> Forward Diffusion (Noise) ──> Decoder Cross-Attention ──> Denoised Patches
                                     (Queries: x_j^{s_j})

Datasets

The authors evaluated TimeDART on 8 standard real-world benchmarks:

  • ETT (ETTh1, ETTh2, ETTm1, ETTm2): Electricity transformer temperatures and loads at 1-hour and 15-minute intervals.
  • Electricity (UCI): Hourly electricity usage across 321 clients.
  • Traffic (PeMS): Hourly road occupancy rates from 862 freeway sensors.
  • Weather (Wetterstation): 21 meteorological indicators sampled every 10 minutes.
  • Exchange: Daily exchange rates across 8 currencies.

No new benchmarks were introduced. Models were tested under both in-domain pre-training and cross-domain pre-training (pre-training on five energy datasets—ETTh1/2, ETTm1/2, Electricity—before fine-tuning on a target dataset).

Key Insights and Ideas

  • Autoregression is vital for SSL transfer: Removing the causal mask from the encoder during ablation resulted in downstream performance worse than random initialization. The causal structure enforces an inductive bias that matches downstream forecasting.
  • Diffusion outperforms MSE for local patch modeling: Rather than predicting deterministic patch averages, the diffusion process learns complex, multimodal intra-patch distributions.
  • Noise scheduling matters more than diffusion steps: Varying total diffusion steps ($T \in \{750, 1000, 1250\}$) showed minor impact, but switching from a linear schedule to a cosine schedule drastically improved performance by providing smoother noise transitions.
  • Shallow decoders protect encoder representations: A single decoder layer outperformed 2- or 3-layer decoders. Allocating too much capacity to the denoising decoder drains parameter utility from the representation encoder.
  • Redundancy dictates patch length: Datasets with high local point-to-point redundancy (e.g., Electricity) benefit from larger patches ($P=8$), while lower redundancy datasets perform better with finer-grained, smaller patches ($P=2$).

Key Results

  • In-Domain State-of-the-Art: TimeDART outperformed top self-supervised baselines (SimMTM, PatchTST-SSL, TimeMAE, CoST) and supervised baselines (PatchTST, DLinear), achieving the best score in 43 out of 64 metric evaluations (~67%) across horizons $H \in \{96, 192, 336, 720\}$.
  • Sweep on ETT Datasets: On ETTh2 and ETTm2, TimeDART achieved the lowest MSE across all prediction horizons (e.g., on ETTm2 at horizon 96, MSE dropped to 0.165 vs. PatchTST's 0.171 and SimMTM's 0.172).
  • Cross-Domain Generalization: Pre-training on mixed energy datasets transferred effectively to target sets, achieving an average MSE of 0.343 on ETTh2 (beating in-domain TimeDART at 0.345 and random init at 0.363).

Limitations and Future Work

  • Sensitivity to Non-Stationary Trends: TimeDART exhibited weaker gains on the financial Exchange dataset, where discrepancies between look-back and forecasting windows and severe trend shifts between validation and test splits provoked overfitting.
  • Patch Size Trade-Off: Smaller patch sizes improve granular fidelity on less redundant data but linearly increase sequence length and quadratic attention cost.
  • Future Work: Developing adaptive patch segmentation and specialized generalization strategies to handle severe distribution shifts in non-stationary datasets.

Summary updated

Abstract

Self-supervised learning has become an essential and popular approach for enhancing time series forecasting, enabling models to learn universal representations from unlabeled data. However, effectively capturing both the global sequence dependence and local detail features within time series data remains challenging. To address this, we propose a novel generative self-supervised method called TimeDART, denoting Diffusion Auto-regressive Transformer for Time series forecasting. In TimeDART, we treat time series patches as basic modeling units. For one thing, we employ an self-attention based Transformer encoder to model the dependencies of inter-patches. For another, we introduce diffusion and denoising mechanisms to capture the locality features of intra-patch. Notably, we design a cross-attention-based denoising decoder that allows for adjustable optimization difficulty in the self-supervised task, facilitating more effective self-supervised pre-training. Extensive experiments demonstrate that TimeDART achieves state-of-the-art fine-tuning performance compared to the most advanced competitive methods in forecasting tasks. Our code is publicly available at https://anonymous.4open.science/r/TimeDART-2024.
BibTeX
@inproceedings{wang2025diffusion,
  author = {Daoyu Wang and Mingyue Cheng and Zhiding Liu and Qi Liu and Enhong Chen},
  title = {{Diffusion Auto-regressive Transformer for Effective Self-supervised Time Series Forecasting}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=v2G9HML7ep},
}
Figure