Abstract
This paper proposes DualVDT, a generative model for multivariate time-series forecasting. By introducing dual reparametrized variational mechanisms into a Variational Autoencoder (VAE), the model achieves a tighter Evidence Lower Bound (ELBO). This mechanism leverages a latent Score-Based Generative Model (SGM) to explicitly denoise perturbations in the latent vector through reverse-time stochastic differential equations and variational ancestral sampling. Additionally, a latent attention mechanism with local-temporal masking is proposed to extract complex dependencies. Experimental results on multiple datasets demonstrate that DualVDT provides superior performance both analytically and empirically compared to state-of-the-art baselines.
Introduction: The Challenge of Error Accumulation
Predicting the future of complex, interconnected systems—such as power grids or epidemiological spreads—requires modeling multivariate time series where variables exhibit both spatial and temporal dependencies. While Variational Autoencoders (VAEs) are powerful tools for capturing underlying data distributions, they often suffer from error accumulation in autoregressive forecasting.
In these scenarios, the Evidence Lower Bound (ELBO) tends to diverge over time, causing the model's internal latent representations to become increasingly noisy. This "latent perturbation" leads to a degradation in long-term forecasting accuracy. Furthermore, capturing the interplay between different variables (factor dependency) and their evolution over time (temporal dependency) simultaneously remains a significant architectural hurdle.
Methodology: DualVDT Architecture
To address these challenges, the authors introduce DualVDT, which stands out by combining the density estimation capabilities of VAEs with the denoising power of Score-Based Generative Models (SGMs).
1. Dual Reparametrized Variational Mechanism
The core innovation is the integration of an SGM into the latent space. Instead of a standard static latent vector, DualVDT treats the latent representation as a point in a diffusion process. A forward Stochastic Differential Equation (SDE) defines how noise is injected, while a reverse-time SDE is employed to "denoise" the latent vector.
This process, coupled with variational ancestral sampling, effectively "tights" the ELBO by minimizing the KL divergence between predicted and actual latent distributions. The posterior of the denoised latent distribution is fused with the dual reparametrized variational density to produce a more accurate representation for the decoder.
2. Local-Temporal Inference
To handle complex dependencies, DualVDT utilizes a latent attention mechanism governed by specialized local-temporal masks. This approach allows the model to treat the input as a dynamic heterogeneous graph, learning both types of interactions simultaneously:
- Spatial Interaction: Captures cross-factor dependencies (e.g., how different sensors in a system influence each other).
- Temporal Interaction: Captures the historical evolution of individual factors.
By applying a binary mask ($\gamma$) within a multi-head self-attention framework, the model extracts multi-level dependencies efficiently, avoiding the need for separate, sequential modules for time and space.
Experimental Evaluation
The model was rigorously tested on two real-world datasets:
- ETDataset (Electricity Transformer Dataset): Focused on predicting oil temperatures in electrical transformers based on various load factors.
- Covid-19 Open Data: Used to model regional virus spread, testing the model's ability to handle complex topological interactions.
Key Results
DualVDT consistently outperformed traditional baselines like LSTMs and Transformers:
- ETTH Dataset: DualVDT achieved a Mean Square Error (MSE) of 0.422, compared to 0.621 for Transformers and 0.955 for LSTMs.
- Covid-19 Dataset: DualVDT reached an MSE of 0.370, significantly better than the Transformer's 0.583.
- Ablation Studies: Tests confirmed that the "Dual Reparametrized" component was the primary driver of performance gains. Enabling this feature consistently reduced error rates regardless of the specific encoder architecture used (FC, CNN, or Local-Temporal).
Discussion and Conclusion
The success of DualVDT stems from treating latent inaccuracy as "noise" that can be systematically removed using the mathematics of diffusion models. By proving that dual reparametrization results in a tighter ELBO, the authors provide a theoretical guarantee for the model's improved generalization.
While effective, the iterative nature of reverse-time SDEs introduces higher computational complexity compared to standard VAEs. Future research may explore optimizing the trade-off between denoising steps and inference speed, or incorporating domain-specific physics into the SDE formulations.
Summary updated