Abstract
QuantileFormer is a novel Transformer-based framework designed for probabilistic time series forecasting. By disentangling smooth temporal trends from complex stochastic noise through a Pattern-Mixture Decomposition approach, the model addresses the challenges of entangled patterns, concept drift, and non-Gaussian distributions. Combining a Variational Autoencoder (VAE) for statistical modeling with a specialized Fusion Transformer, QuantileFormer achieves state-of-the-art accuracy and superior calibration of prediction intervals across diverse real-world datasets.
Introduction
Traditional time series forecasting primarily focuses on point-wise predictions. However, in critical domains such as renewable energy management and healthcare, understanding the underlying uncertainty is paramount. Probabilistic forecasting aims to predict the entire distribution of future outcomes rather than a single value.
Current models face three significant hurdles:
- Entangled Patterns: Real-world data often conflates long-term trends with high-frequency fluctuations.
- Concept Drift: The statistical properties of time series data frequently shift over time.
- Complex Distributions: Empirical data rarely follows simple parametric distributions, often manifesting as a mixture of various underlying distributions.
Methodology
The core of QuantileFormer is its Pattern-Mixture Decomposition strategy, which segments the forecasting task into manageable sub-problems.
Drift-Divergence Decomposition
The framework employs a "quantile filter" to bifurcate the input series into two components: Quantile Drift, representing the smooth underlying trend, and Divergence Patterns, capturing the complex residual noise.
Gaussian Mixture Decomposition
To handle the complexity of the divergence patterns, the model decomposes them into a mixture of Gaussian distributions. This allows for a more nuanced representation of the uncertainty "shape."
Architecture
The QuantileFormer architecture integrates three primary modules:
- Transformer Encoder: Captures long-range temporal dependencies within the "Drift" component.
- VAE Network: Utilizes variational inference on Gaussian mixture parameters to extract global statistical features, allowing the model to adapt to varying probabilistic characteristics.
- Fusion Transformer: Employs cross-attention to merge temporal features from the encoder with statistical insights from the VAE, generating final multi-quantile predictions.
Experimental Setup
The model was rigorously evaluated against six diverse real-world benchmarks representing various industrial and environmental sectors.
The datasets include electricity consumption, transformer temperature (ETT), traffic occupancy rates, and solar/wind power production data.
Results and Discussion
QuantileFormer consistently outperformed established baselines, including DeepAR, TFT, and MQRNN, as well as adapted point-prediction models like iTransformer.
Performance Metrics
The model achieved an average $q$-risk reduction of 14% to 27% across different quantile levels. Furthermore, the authors introduced a new metric, cpaw (Coverage Probability with Normalized Averaged Width), to evaluate the reliability and precision of prediction intervals. QuantileFormer demonstrated a 51% improvement in cpaw on the Traffic dataset compared to other Transformer-based models.
Key Insights
- Disentanglement: Separating trend modeling (Transformer) from distribution modeling (VAE) significantly enhances performance.
- Adaptive Uncertainty: The VAE-based latent posterior allows the model to dynamically adjust to concept drift within specific time windows.
Conclusion
QuantileFormer represents a significant advancement in probabilistic time series forecasting. By leveraging pattern-mixture decomposition and a hybrid VAE-Transformer architecture, it provides more accurate and better-calibrated uncertainty estimates. Future work will explore extending this framework to multivariate forecasting to capture cross-correlations in high-dimensional data.
Summary updated