Heteroscedastic Temporal Variational Autoencoder For Irregularly Sampled Time Series

Satya Narayan Shukla, Benjamin MarlinICLR 2022 · OpenReview

TL;DR

The Heteroscedastic Temporal Variational Autoencoder (HeTVAE) is a novel deep learning framework designed to interpolate irregularly sampled time series while accurately quantifying uncertainty. By combining a sparsity-aware attention mechanism with a heteroscedastic output layer, it outperforms existing models in predicting both the values of missing data and how much we should trust those predictions.

What Problem Does This Paper Address?

In fields like healthcare (ICU monitoring) or climate science, data rarely arrives on a clean, regular heartbeat. Sensors fail, patients are moved for tests, and different variables are recorded at different frequencies. This results in irregularly sampled multivariate time series.

Standard deep learning models struggle here because they typically expect fixed-size, regularly spaced inputs. While recent Variational Autoencoders (VAEs) have made progress, they often suffer from two critical flaws:

  1. Normalization Bias: They often "normalize away" the information about how sparse the data is, treating a 10-hour gap the same as a 10-minute gap.
  2. Homoscedasticity: They assume the level of noise or uncertainty is constant across all time points. In reality, we should be much more uncertain about a patient's heart rate during a long data gap than immediately after a fresh measurement.

Methods

To address these issues, the authors introduce HeTVAE, which features several key innovations:

HeTVAE Architecture
  • UnTAN (Uncertainty Aware Multi-Time Attention Network): This is a new input layer that splits the signal into two pathways: an Intensity (INT) pathway that encodes the sparsity of observations, and a Value (VAL) pathway that encodes the actual data values. This ensures the model explicitly knows where the gaps are.
  • Hybrid Latent Pathways: The architecture uses both probabilistic (sampled) and deterministic latent paths to propagate information, allowing for a more flexible representation of the time series.
  • Heteroscedastic Output Layer: Unlike standard VAEs that output a single mean and a fixed variance, HeTVAE outputs a time-varying mean and variance. This allows the "uncertainty envelope" to grow and shrink based on data density.
  • Augmented Training Objective: Training heteroscedastic models is notoriously difficult because the model can fall into a "local optimum" where it simply predicts high variance everywhere to explain away its errors. The authors add an "uncertainty-agnostic" squared loss term to the standard ELBO (Evidence Lower Bound) to force the model to learn the underlying structure of the data before fine-tuning its uncertainty estimates.

Datasets

The model was evaluated on a mix of synthetic and high-stakes real-world datasets:

  • PhysioNet Challenge 2012: Sparse and irregular ICU records (37 physiological variables).
  • MIMIC-III: A massive dataset of clinical ICU data (12 variables used for interpolation).
  • USHCN Climate Dataset: Daily weather measurements from 1910–2009, processed into multi-rate time series.
  • UCI Electricity Dataset: Household power consumption data.
  • Synthetic Dataset: Specifically designed to visualize how uncertainty behaves in the presence of varying observation gaps.

Key Insights and Ideas

  • Sparsity is a Signal: The "intensity" of observations is a critical feature. By explicitly modeling how often data appears, the model learns that gaps naturally imply higher uncertainty.
  • The "Cheating" Problem: The paper highlights a common failure mode in probabilistic deep learning: models "giving up" on the mean and using high variance as a crutch. Their augmented loss function is a practical "nudge" that ensures the model actually learns to track the data.
  • Deep Learning vs. Gaussian Processes (GPs): GPs are the gold standard for uncertainty but are computationally expensive ($O(N^3)$) and hard to scale to multivariate data. HeTVAE offers GP-like uncertainty behavior (widening in gaps, narrowing at points) with the scalability and speed of a neural network.

Key Results

HeTVAE consistently outperformed baselines—including Gaussian Process Regression (GPR), ODE-based VAEs, and the previous state-of-the-art mTAN—primarily in Negative Log Likelihood (NLL), which measures how well a model captures the true distribution of the data.

  • PhysioNet: HeTVAE achieved an NLL of 0.5542, a massive improvement over the previous best VAE-based model (HTVAE mTAN) at 1.2426.
  • MIMIC-III: HeTVAE led with an NLL of 0.6662 vs. the next best at 0.8360.
Qualitative comparison of uncertainty estimates
  • Qualitative Success: Visualizations on synthetic data show that HeTVAE’s uncertainty (the shaded confidence region) correctly expands during data gaps and contracts when observations are present. In contrast, previous deep learning models (like mTAN) maintained a nearly constant, "overconfident" uncertainty level, failing to reflect the lack of data.

Limitations and Future Work

  • Marginal vs. Joint Distributions: Currently, HeTVAE produces marginal distributions for each time point. This means that while it knows the range of likely values at any given second, it doesn't necessarily guarantee that a continuous sample drawn from the model will be a smooth, physically realistic trajectory.
  • Residual Correlations: The authors suggest that future work could involve accounting for correlations in the output layer to produce smoother, more coherent sampled trajectories, similar to what Gaussian Processes provide out of the box.

Summary updated

Abstract

Irregularly sampled time series commonly occur in several domains where they present a significant challenge to standard deep learning models. In this paper, we propose a new deep learning framework for probabilistic interpolation of irregularly sampled time series that we call the Heteroscedastic Temporal Variational Autoencoder (HeTVAE). HeTVAE includes a novel input layer to encode information about input observation sparsity, a temporal VAE architecture to propagate uncertainty due to input sparsity, and a heteroscedastic output layer to enable variable uncertainty in the output interpolations. Our results show that the proposed architecture is better able to reflect variable uncertainty through time due to sparse and irregular sampling than a range of baseline and traditional models, as well as recently proposed deep latent variable models that use homoscedastic output layers.
BibTeX
@inproceedings{shukla2022heteroscedastic,
  author = {Satya Narayan Shukla and Benjamin Marlin},
  title = {{Heteroscedastic Temporal Variational Autoencoder For Irregularly Sampled Time Series}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=Az7opqbQE-3},
}
Figure