Cumulative Difference Learning VAE for Time-Series with Temporally Correlated Inflow-Outflow

Tianchun Li, Chengxiang Wu, Pengyi Shi, Xiaoqian WangAAAI 2024 · DOI

TL;DR

This paper introduces DT-VAE, a novel Variational Autoencoder (VAE) framework for time-series generation that tackles the critical issue of error accumulation in existing methods. By modeling time series as a function of inflow and outflow variables and learning cumulative differences, DT-VAE significantly improves the accuracy and reliability of generated synthetic data, particularly for downstream decision-making tasks.

What Problem Does This Paper Address?

Generating realistic and accurate synthetic time-series data is crucial for various applications, from augmenting datasets for predictive modeling to simulating scenarios for decision-making. However, existing generative models often suffer from two major limitations:

  1. Error Accumulation: Many methods generate time series autoregressively, meaning errors made at earlier time steps propagate and amplify over time, leading to generated sequences that deviate significantly from the true underlying trend. This is particularly problematic for long-term forecasting or scenario generation.
  2. Lack of Domain Knowledge and Interpretability: General-purpose generative models often ignore the inherent structure of time-series data and lack theoretical justification, making them less effective and harder to interpret. For instance, many real-world time series, like hospital census data, are influenced by unobserved factors like patient arrivals (inflow) and departures (outflow), which are often correlated but not directly recorded.

The authors are motivated by the need for accurate synthetic hospital census data, especially in the context of the COVID-19 pandemic, for critical decision-making like bed capacity planning and nurse staffing.

Methods

The core innovation of this paper is the Cumulative Difference Learning mechanism integrated into a VAE framework, leading to the DT-VAE model.

  • Problem Formulation: The authors model observed time series $X_t$ as being driven by the previous observation $X_{t-1}$, an "inflow" variable $A_t$, and an "outflow" variable $D_t$, such that $X_t = X_{t-1} + A_t - D_t + \epsilon$. Crucially, $A_t$ and $D_t$ are further driven by unobserved "environmental factors" $Z_t$.
  • Cumulative Difference Learning: To combat error accumulation, they transform the problem by defining a cumulative difference variable $\Gamma_t = \sum_{i=1}^t (A_i - D_i)$. This allows them to express the observed variable $X_t$ as $X_t = X_0 + \Gamma_t + \epsilon_t$, where $X_0$ is the initial state. This formulation means $X_t$ depends on the cumulative difference $\Gamma_t$ and the initial state $X_0$, rather than relying on a step-by-step reconstruction from $X_{t-1}$. This bypasses the error propagation issue.
  • DT-VAE Architecture: They build a VAE around this formulation.
  • Encoder: A recurrent neural network (RNN) based encoder learns to map observed cumulative differences $\Gamma_{1:T}$ to a posterior distribution over latent variables $z_{1:T}$. This encoder captures historical information from both latent variables and observed data.
  • Decoder: A recurrent decoder learns to map latent variables $z_{1:t}$ to reconstruct the cumulative difference $\Gamma_t$. This decoder is designed to avoid error accumulation by directly generating $\Gamma_t$ from latent variables.
  • DT-VAE-GAN: For enhanced generation quality, they also propose integrating DT-VAE with a Generative Adversarial Network (GAN) architecture.

Datasets

The authors evaluated their methods on a variety of datasets:

  • Semi-Synthetic Data: Generated to simulate hospital census data, allowing for direct comparison with known ground truth.
  • COVID-19 Hospitalization Datasets:
    • County-DH: Newly curated datasets for COVID-19 hospitalizations at the county level from California, New York, and Pennsylvania.
  • States-DH: Newly curated datasets for COVID-19 hospitalizations at the state level from the US.
  • Public Benchmark Datasets:
    • Energy: Hourly solar energy collection data.
  • Traffic: Hourly occupancy rates from traffic management data.
  • Stock Market: Daily stock data.

Key Insights and Ideas

  • The Power of Cumulative Differences: The core insight is that by reframing the problem to learn cumulative differences rather than direct step-by-step predictions, error accumulation can be effectively mitigated. This is a significant departure from traditional autoregressive models.
  • Incorporating Domain Knowledge: The model explicitly incorporates the inflow/outflow structure, which is common in many real-world time-series phenomena. This domain-aware approach leads to more interpretable and theoretically grounded generation.
  • Theoretical Justification: The VAE framework provides a mathematically sound basis for the generative process, offering interpretability and a clear objective function (ELBO) for training.
  • Novel Evaluation Metric (DTP): The introduction of the Diverse Trend Preservation (DTP) evaluation metric is a valuable contribution. It moves beyond simple statistical similarity (like FID scores) to assess how well generated data preserves the distinct trends present in the original data, offering a more visually interpretable assessment.

Key Results

  • Superior Trend Preservation: DT-VAE and DT-VAE-GAN consistently outperformed other benchmarks (TimeGAN, PSA-GAN, T-VAE-GAN, T-VAE) in preserving temporal trends, as evidenced by both visual inspection (Figure 4) and the novel DTP scores (Table 1). DT-VAE showed up to a 30% improvement in DTP scores.
  • Reduced Error Accumulation: The generated time series from DT-VAE showed significantly fewer chaotic shapes and deviations from the original trend compared to PSA-GAN and TimeGAN, particularly in scenarios with distinct trends (Figure 1).
  • Improved Long-Term Prediction: In "Train on Synthetic, Test on Real" (TSTR) experiments for long-term prediction (Table 2), DT-VAE and DT-VAE-GAN achieved substantial improvements in Mean Squared Error (MSE), with up to 65% reduction compared to baselines.
  • Effective on Diverse Domains: The model demonstrated strong performance across various domains, including healthcare, energy, traffic, and stock markets, highlighting its generalizability.
  • t-SNE Visualization: t-SNE plots (Figure 5) showed that DT-VAE's generated data clusters more closely with the original data compared to some benchmarks, suggesting better representation of the underlying data distribution.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, VAEs and GANs can be computationally intensive to train. The paper mentions integrating DT-VAE with VAE-GAN for improved accuracy, which might further increase computational demands.
  • Generalizability of Inflow/Outflow Assumption: The core formulation relies on the inflow/outflow structure. While the paper argues for its broad applicability, there might be time-series phenomena where this assumption is less fitting, potentially impacting interpretability.
  • Future Work: The authors suggest exploring more complex architectures for the encoder and decoder, such as transformers, to further enhance the model's capabilities. They also hint at the potential for DT-VAE to be applied to other domains where similar inflow/outflow dynamics exist. The DT-VAE-GAN integration is also presented as a path for improved generative accuracy.

Summary updated

BibTeX
@inproceedings{li2024cumulative,
  author = {Tianchun Li and Chengxiang Wu and Pengyi Shi and Xiaoqian Wang},
  title = {{Cumulative Difference Learning VAE for Time-Series with Temporally Correlated Inflow-Outflow}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i12.29266},
}
Figure