Probabilistic Transformer For Time Series Analysis

Binh Tang, David S. MattesonNeurIPS 2021

Generative modeling of multivariate time series—predicting how multiple related variables change over time—is notoriously difficult. Traditional State-Space Models (SSMs) often rely on the "Markov property," assuming the next state depends only on the current one, which fails to capture complex, long-distance relationships. Conversely, standard Transformers are typically deterministic and autoregressive. In long-term forecasting, small errors in early steps can snowball, leading to "drift" and a failure to account for multiple plausible futures.

To address these challenges, researchers have introduced ProTran (Probabilistic Transformer), a novel architecture that marries the long-range dependency modeling of Transformers with the principled uncertainty handling of SSMs. By replacing traditional recurrent transitions with attention mechanisms, ProTran enables non-autoregressive, diverse, and highly accurate forecasting for complex multivariate data like traffic patterns and human motion.

The ProTran Architecture

The core contribution of this work is treating the Transformer's internal mechanism as a transition function within a State-Space Model. The authors aim to create a model that can "look back" across the entire history without being limited by Markovian assumptions, while still providing a probabilistic framework to quantify uncertainty.

Graphical models comparing LDS and ProTran architectures

Key innovations in the ProTran architecture include:

  • Attention-based Transitions: Instead of using Recurrent Neural Networks (RNNs) or linear transitions, ProTran uses self-attention between latent variables. This breaks the Markov chain, allowing the state at time $t$ to depend directly on any previous state in the sequence and focus on historical patterns that matter.
  • Deterministic + Stochastic Hybrid: The model maintains a deterministic hidden representation to reliably carry information across time, while incorporating stochastic latent variables ($z_t$) at each time step. This turns the Transformer into a generative model capable of producing a distribution of possible futures rather than a single point estimate.
  • Hierarchical Structure: For complex data, the model is extended to multiple layers of stochastic variables (as seen in the generation and inference models above), increasing expressiveness without losing the ability to perform tractable inference.
  • Smoothing via Attention: To train the model, the researchers use a variational approach. The inference procedure acts like a deep learning version of a classical smoother, using bidirectional attention to look at both past and future observations to refine the estimate of the latent states.
  • Non-Autoregressive Generation: Unlike many time-series Transformers, ProTran generates future steps in the latent space without needing to feed high-dimensional observations back into the model, reducing computational cost and error accumulation.

Experimental Results

The authors evaluated ProTran on two distinct types of tasks: multivariate time series forecasting (using SOLAR, ELECTRICITY, TRAFFIC, TAXI, and WIKIPEDIA datasets) and human motion prediction (using Human3.6M and HumanEva-I datasets).

CRPS evaluation on time series datasets
  • State-of-the-Art Accuracy: As shown in the results table, ProTran significantly outperformed competitive baselines like DeepAR, GP-Copula, and TimeGrad in terms of Continuous Ranked Probability Score (CRPS)—a metric that measures the quality of probabilistic forecasts. It achieved particularly strong SOTA results on the TRAFFIC and SOLAR datasets.
  • Superior Motion Prediction: In human motion tasks, ProTran achieved the lowest Average Displacement Error (ADE) and Final Displacement Error (FDE). On the large-scale Human3.6M dataset, it achieved an ADE of 0.381, beating the previous best (DLow) at 0.425.
  • Uncertainty Calibration: Qualitative results demonstrated that the model’s prediction intervals naturally widen as the forecast moves further into the future or encounters high-magnitude observations, showing a realistic grasp of uncertainty.

Limitations and Future Directions

While highly effective, ProTran shares a common limitation with most standard Transformers: it suffers from $O(T^2)$ time and memory complexity relative to the sequence length $T$. This makes it difficult to apply to extremely long sequences without using "Sparse Transformer" variants. Additionally, while the attention mechanism is parallelizable, the recursive nature of the latent dynamics in this specific SSM formulation prevents the full training speedups usually associated with Transformers.

Future work may explore integrating sparse attention mechanisms to handle longer time horizons and applying the model to healthcare domains, such as tracking disease progression over time.

Summary updated

BibTeX
@inproceedings{tang2021probabilistic,
  author = {Binh Tang and David S. Matteson},
  title = {{Probabilistic Transformer For Time Series Analysis}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
}
Figure