QuantileFormer: Probabilistic Time Series Forecasting with a Pattern-Mixture Decomposed VAE Transformer

Yimiao Shao, Wenzhong Li, Kang Xia, Kaijie Lin, Mingkai Lin, Sanglu Lu

Abstract

QuantileFormer is a novel Transformer-based framework designed for probabilistic time series forecasting. By disentangling smooth temporal trends from complex stochastic noise through a Pattern-Mixture Decomposition approach, the model addresses the challenges of entangled patterns, concept drift, and non-Gaussian distributions. Combining a Variational Autoencoder (VAE) for statistical modeling with a specialized Fusion Transformer, QuantileFormer achieves state-of-the-art accuracy and superior calibration of prediction intervals across diverse real-world datasets.

Introduction

Traditional time series forecasting primarily focuses on point-wise predictions. However, in critical domains such as renewable energy management and healthcare, understanding the underlying uncertainty is paramount. Probabilistic forecasting aims to predict the entire distribution of future outcomes rather than a single value.

Current models face three significant hurdles:

  1. Entangled Patterns: Real-world data often conflates long-term trends with high-frequency fluctuations.
  2. Concept Drift: The statistical properties of time series data frequently shift over time.
  3. Complex Distributions: Empirical data rarely follows simple parametric distributions, often manifesting as a mixture of various underlying distributions.

Methodology

The core of QuantileFormer is its Pattern-Mixture Decomposition strategy, which segments the forecasting task into manageable sub-problems.

Drift-Divergence Decomposition

The framework employs a "quantile filter" to bifurcate the input series into two components: Quantile Drift, representing the smooth underlying trend, and Divergence Patterns, capturing the complex residual noise.

Gaussian Mixture Decomposition

To handle the complexity of the divergence patterns, the model decomposes them into a mixture of Gaussian distributions. This allows for a more nuanced representation of the uncertainty "shape."

Decomposition of complex stochastic noise into a mixture of Gaussian distributions.

Architecture

The QuantileFormer architecture integrates three primary modules:

  • Transformer Encoder: Captures long-range temporal dependencies within the "Drift" component.
  • VAE Network: Utilizes variational inference on Gaussian mixture parameters to extract global statistical features, allowing the model to adapt to varying probabilistic characteristics.
  • Fusion Transformer: Employs cross-attention to merge temporal features from the encoder with statistical insights from the VAE, generating final multi-quantile predictions.

Experimental Setup

The model was rigorously evaluated against six diverse real-world benchmarks representing various industrial and environmental sectors.

Diverse application domains and datasets used for evaluation, including solar, wind, and traffic data.

The datasets include electricity consumption, transformer temperature (ETT), traffic occupancy rates, and solar/wind power production data.

Results and Discussion

QuantileFormer consistently outperformed established baselines, including DeepAR, TFT, and MQRNN, as well as adapted point-prediction models like iTransformer.

Performance Metrics

The model achieved an average $q$-risk reduction of 14% to 27% across different quantile levels. Furthermore, the authors introduced a new metric, cpaw (Coverage Probability with Normalized Averaged Width), to evaluate the reliability and precision of prediction intervals. QuantileFormer demonstrated a 51% improvement in cpaw on the Traffic dataset compared to other Transformer-based models.

Key Insights

  • Disentanglement: Separating trend modeling (Transformer) from distribution modeling (VAE) significantly enhances performance.
  • Adaptive Uncertainty: The VAE-based latent posterior allows the model to dynamically adjust to concept drift within specific time windows.

Conclusion

QuantileFormer represents a significant advancement in probabilistic time series forecasting. By leveraging pattern-mixture decomposition and a hybrid VAE-Transformer architecture, it provides more accurate and better-calibrated uncertainty estimates. Future work will explore extending this framework to multivariate forecasting to capture cross-correlations in high-dimensional data.

Summary updated

Abstract

Probabilistic time series forecasting has attracted an increasing attention in machine learning community for its potential applications in the fields of renewable energy, traffic management, healthcare, etc. Previous research mainly focused on extracting long-range dependencies for point-wise prediction, which fail to capture complex temporal patterns and statistical characteristics for probabilistic analysis. In this paper, we propose a novel pattern-mixture decomposition method that decomposes long-term series into quantile drift, divergence patterns, and Gaussian mixture components, which can effectively capture the intricate temporal patterns and stochastic characteristics in time series. Based on pattern-mixture decomposition, we propose a novel Transformer-based model called QuantileFormer for probabilistic time series forecasting. It takes the the comprehensive drift-divergence mixture patterns as features, and designs a variational inference based fusion Transformer architecture to generate quantile prediction results. Extensive experiments show that the proposed method consistently boosts the baseline methods by a large margin and achieves state-of-the-art performance on six real-world benchmarks.
BibTeX
@inproceedings{shao2025quantileformer,
  author = {Yimiao Shao and Wenzhong Li and Kang Xia and Kaijie Lin and Mingkai Lin and Sanglu Lu},
  title = {{QuantileFormer: Probabilistic Time Series Forecasting with a Pattern-Mixture Decomposed VAE Transformer}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/684},
}
Figure