TL;DR
TimeDART introduces a generative self-supervised pre-training framework that pairs an auto-regressive Transformer encoder with a diffusion-based denoising patch decoder for time series forecasting. By using causal modeling to capture global inter-patch trends and diffusion to recover fine-grained intra-patch dynamics, the model substantially narrows the gap between self-supervised pre-training and downstream forecasting. Across extensive benchmarks, TimeDART achieves state-of-the-art transfer performance compared to existing contrastive, masked autoencoder, and supervised baselines.
What Problem Does This Paper Address?
Self-supervised learning (SSL) in time series usually relies on one of two strategies:
- Masked Autoencoders (MAE): Reconstruct masked values, but create a distribution gap between pre-training (corrupted data) and downstream forecasting (full clean sequences).
- Contrastive Learning: Learn discriminative features by comparing similar/dissimilar windows, which is ill-suited for time series where defining pair similarity is ambiguous and temporal generative dynamics matter most.
Moreover, standard Mean Squared Error (MSE) pre-training encourages overly smooth predictions that miss multimodal, localized noise. Time series models need to jointly capture long-range temporal trends (global context) and local high-frequency variations (intra-patch details) without degrading downstream transferability.
Methods
TimeDART breaks the look-back window into non-overlapping patches (setting stride $S = \text{patch length } P$) under a channel-independent setting. The architecture couples two complementary generative mechanisms:
- Causal Transformer Encoder (Inter-Patch Modeling): Processes clean patch embeddings using a causal mask and a start-of-sequence (
SOS) token. This forces the encoder to model global sequence dependencies auto-regressively—aligning pre-training directly with the "past predicts future" forecasting paradigm. - Diffusion Denoising Decoder (Intra-Patch Modeling): Adds forward Gaussian noise to each patch independently across time steps using a smooth cosine noise schedule. A cross-attention decoder then takes the noisy patches as queries and the causal encoder states as keys and values. A diagonal mask ensures the $j$-th noisy patch only attends to encoder representations aggregated up to time step $j-1$.
- Generative Objective: Pre-training minimizes the diffusion Evidence Lower Bound (ELBO), optimizing the model to denoise corrupted patches conditioned on past context.
- Downstream Fine-Tuning: The diffusion decoder is removed, and a simple linear projection head is attached to the frozen or fine-tuned Transformer encoder to produce multi-step forecasts via one-step generation.
Input Patches [x_1, ..., x_N]
├──> Causal Transformer Encoder ──> Contextual States f(z_{1:j-1})
│ │ (Keys & Values)
└──> Forward Diffusion (Noise) ──> Decoder Cross-Attention ──> Denoised Patches
(Queries: x_j^{s_j})
Datasets
The authors evaluated TimeDART on 8 standard real-world benchmarks:
- ETT (ETTh1, ETTh2, ETTm1, ETTm2): Electricity transformer temperatures and loads at 1-hour and 15-minute intervals.
- Electricity (UCI): Hourly electricity usage across 321 clients.
- Traffic (PeMS): Hourly road occupancy rates from 862 freeway sensors.
- Weather (Wetterstation): 21 meteorological indicators sampled every 10 minutes.
- Exchange: Daily exchange rates across 8 currencies.
No new benchmarks were introduced. Models were tested under both in-domain pre-training and cross-domain pre-training (pre-training on five energy datasets—ETTh1/2, ETTm1/2, Electricity—before fine-tuning on a target dataset).
Key Insights and Ideas
- Autoregression is vital for SSL transfer: Removing the causal mask from the encoder during ablation resulted in downstream performance worse than random initialization. The causal structure enforces an inductive bias that matches downstream forecasting.
- Diffusion outperforms MSE for local patch modeling: Rather than predicting deterministic patch averages, the diffusion process learns complex, multimodal intra-patch distributions.
- Noise scheduling matters more than diffusion steps: Varying total diffusion steps ($T \in \{750, 1000, 1250\}$) showed minor impact, but switching from a linear schedule to a cosine schedule drastically improved performance by providing smoother noise transitions.
- Shallow decoders protect encoder representations: A single decoder layer outperformed 2- or 3-layer decoders. Allocating too much capacity to the denoising decoder drains parameter utility from the representation encoder.
- Redundancy dictates patch length: Datasets with high local point-to-point redundancy (e.g., Electricity) benefit from larger patches ($P=8$), while lower redundancy datasets perform better with finer-grained, smaller patches ($P=2$).
Key Results
- In-Domain State-of-the-Art: TimeDART outperformed top self-supervised baselines (SimMTM, PatchTST-SSL, TimeMAE, CoST) and supervised baselines (PatchTST, DLinear), achieving the best score in 43 out of 64 metric evaluations (~67%) across horizons $H \in \{96, 192, 336, 720\}$.
- Sweep on ETT Datasets: On ETTh2 and ETTm2, TimeDART achieved the lowest MSE across all prediction horizons (e.g., on ETTm2 at horizon 96, MSE dropped to 0.165 vs. PatchTST's 0.171 and SimMTM's 0.172).
- Cross-Domain Generalization: Pre-training on mixed energy datasets transferred effectively to target sets, achieving an average MSE of 0.343 on ETTh2 (beating in-domain TimeDART at 0.345 and random init at 0.363).
Limitations and Future Work
- Sensitivity to Non-Stationary Trends: TimeDART exhibited weaker gains on the financial Exchange dataset, where discrepancies between look-back and forecasting windows and severe trend shifts between validation and test splits provoked overfitting.
- Patch Size Trade-Off: Smaller patch sizes improve granular fidelity on less redundant data but linearly increase sequence length and quadratic attention cost.
- Future Work: Developing adaptive patch segmentation and specialized generalization strategies to handle severe distribution shifts in non-stationary datasets.
Summary updated