Calibrated Uncertainty Under Sensor Dropout in Multimodal Physiological VAEs

Wearable and clinical monitors combine several body signals: brain activity (EEG), heart activity (ECG, or PPG, the optical pulse signal), skin conductance (EDA, a readout of sweat-gland arousal), and breathing. In practice some of these sensors fail or get unplugged. Current multimodal models deal with missing sensors by training to fill the gap and then producing an answer anyway. What they do not report is how much the missing sensor should have mattered. A sleep-staging model that loses EEG is a different instrument from one that loses breathing effort, and a clinician should be told so.

We propose a variational autoencoder (VAE), a model that compresses each input into a probability distribution over a small set of hidden variables rather than a single point. Its hidden state is split into a part shared across sensors and a part private to each sensor. The model is trained so that the width of the shared distribution honestly tracks how much the remaining sensors constrain the physiological state. The claim we want to test is narrow. Under any pattern of missing sensors, the posterior width should predict the actual increase in downstream error well enough to support abstention and calibrated imputation. A deterministic masked autoencoder of the same size, even one wrapped in an ensemble, should do this measurably worse. We do not claim better accuracy when all sensors are present.

Why this question, and what already exists

The closest generative work splits the field into pieces that have not been combined. Promoting cross-modal representations to improve multimodal foundation models for physiological signals trains a deterministic masked autoencoder on PhysioNet 2018 sleep recordings. It randomly drops whole modalities, which forces cross-modal reconstruction, raises an aggregate downstream score to 1.144, and spreads reliance more evenly across sensors. This paper motivates our work and is also our strongest baseline. It shows that missing-sensor training helps, but its point reconstructions carry no statement of confidence.

Private-Shared Disentangled Multimodal VAE for Learning of Latent Representations supplies the method we reuse: a shared latent fused across modalities by a product of experts, private latents per modality, and cross-modal reconstruction. It was only ever tested on static image and text pairs sampled at a single rate.

Two papers show that probabilistic latents help with one fixed missing pair, in other settings. Robust Multimodal Survival Prediction with Conditional Latent Differentiation Variational AutoEncoder aligns an image-derived posterior to a missing genomics posterior for static data. Unlocking Cross-Modal Biosignal Synthesis: A Temporally-Aware VAE-Diffusion Model generates heart sounds from ECG in one direction only. That paper's explicit modeling of the electrical-to-mechanical delay also informs our design.

The literature also contains a result that pushes against us. SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals argues that contrastive alignment beats reconstruction because reconstruction wastes capacity on raw signal detail. We take that argument seriously. It is why we do not stake the proposal on downstream accuracy.

Core Claim

The physiological literature describes two specific difficulties. First, different sensors carry very different amounts of information: EEG dominates sleep staging and nasal pressure dominates apnea detection. Second, autonomic responses lag cortical events by hundreds of milliseconds to seconds.

We claim that a shared-plus-private VAE handles both if it is built as follows. Its encoders run at each sensor's native sampling rate. Its shared latent is a slow sequence. Each modality's decoder reads that shared sequence through a learned delay kernel. It is trained over random subsets of sensors. Under these conditions, the uncertainty in its shared posterior becomes an accurate, per-subset estimate of lost information. Two signatures would confirm this. Across missing-sensor patterns, the ranking by posterior entropy should match the ranking by observed error increase. Within a recording, a present but artifact-corrupted sensor should widen the posterior, rather than being treated as fully trustworthy.

Research Program

The model has one 1D convolutional encoder per modality, each operating at native rate (EEG at hundreds of Hz, EDA at a few Hz). Each encoder outputs two things:

  • a Gaussian "expert" over a shared latent sequence $z_s(t)$ at one step per 30-second epoch (or per second for stress data);
  • a private latent sequence $z_p^{m}(t)$ for its own modality $m$.

Each modality decoder reconstructs its signal from $z_p^m$ together with $z_s$ convolved with a learned causal kernel $k_m$. The kernel lets EDA at time $t$ depend on shared state from earlier times.

Training samples a random observed subset $S$ of modalities and maximizes

$$\mathcal{L} = \mathbb{E}_{S}\Big[\mathbb{E}_{q(z \mid x_S)} \sum_{m=1}^{M} \log p(x_m \mid k_m * z_s, z_p^m) - \mathrm{KL}\big(q(z \mid x_S)\,\|\,p(z)\big)\Big] + \lambda\, \ell_{\text{task}}.$$

The terms are defined as follows:

  • $x_S$ is the set of observed signals.
  • $q(z \mid x_S)$ is the fused posterior over shared and private latents. For a missing modality $m$, $z_p^m$ is drawn from the prior.
  • The sum runs over all $M$ modalities, so missing ones must be reconstructed.
  • $\ell_{\text{task}}$ is an optional supervised head on $z_s$, weighted by $\lambda$.

There is a known risk here. A plain product of experts adds precisions, so its variance shrinks with the number of sensors present regardless of what those sensors actually say. That would defeat the claim. We therefore compare three fusion rules: plain product, mixture of experts, and a product with a learned per-expert precision scale fitted on held-out data. We report which fusion rule, if any, yields honest widths.

Downstream use is deliberately simple. A linear probe reads the mean of $z_s$. Abstention thresholds on mean posterior entropy are set on validation subjects. Imputed waveforms are drawn as posterior samples rather than point estimates.

Baselines

The naive reference point is late fusion: one classifier per modality, with predictions averaged over whichever modalities are present. This handles missing sensors at no cost. StressID: a Multimodal Dataset for Stress Identification reports an F1 of 0.72 for two-class stress with average-rule fusion. On StressID we reproduce that protocol and use that figure as the floor. Its confidence is the averaged class probability, after temperature scaling per missing-sensor pattern on validation data.

The strongest existing method is the masked autoencoder with modality drop from Promoting cross-modal representations to improve multimodal foundation models for physiological signals, which reports an aggregate score of 1.144 on PhysioNet 2018. Our implementation must first reproduce roughly that score under the paper's protocol before any comparison counts. The setup is then matched in four ways: the VAE shares the same convolutional tokenizers, a matched parameter count, the same training hours, and the same hyperparameter search budget. Its uncertainty comes from temperature-scaled probe outputs per missing pattern.

Because a single deterministic model is an easy target, we also run a five-member ensemble of that autoencoder. The ensemble's spread is the most credible non-variational uncertainty estimate. We report it at both equal total compute and fivefold compute.

SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals reports a macro AUROC of 0.906 for sleep staging on its own Stanford data, which is not our benchmark. We retrain its leave-one-out contrastive objective on PhysioNet 2018 with the same backbones and compare it under missing sensors.

An accuracy upper bound comes from training a separate supervised classifier for every one of the fifteen subsets of four PhysioNet modality groups.

Three ablations isolate the key ingredients:

  • Deterministic twin: the same architecture with the KL term removed and uncertainty taken from an ensemble. This isolates the effect of the variational posterior.
  • No delay kernel: this isolates the effect of lag modeling.
  • No private latents: this tests whether sensor-specific noise leaks into $z_s$ and inflates or deflates its width.

Beating the baselines means the following, judged against the strongest uncertainty baseline (the ensemble). We require higher Spearman correlation between per-pattern entropy and per-pattern error increase, lower expected calibration error averaged over patterns, and better selective risk at 80% coverage. Each must hold with subject-level bootstrap confidence intervals that exclude zero, while full-sensor accuracy stays within one standard error of the masked autoencoder.

Why not just…?

Why not just use late fusion, which already copes with missing sensors? Late fusion's confidence comes from averaging per-modality classifiers. It cannot represent that EEG and breathing carry overlapping information, so losing one may cost little when the other is present. Classification of Perceived Human Stress using Physiological Signals found classical per-model classifiers far below an integrated network (50% for SVM versus 75% for the MLP). That gap is weak evidence, since the cohort is small. The per-pattern calibration comparison above settles the question directly.

Why not just ensemble the masked autoencoder? This is the real threat. Ensembles are a strong uncertainty baseline, and nothing in the file shows a VAE beating one on physiological data. If the ensemble matches our per-pattern rank correlation and calibration at fivefold compute, the honest conclusion is that the variational posterior buys only cost savings. We will report it that way. If it matches at equal compute, the proposal's claim fails.

Why not just train contrastive embeddings and threshold their similarity? Contrastive alignment gives a similarity score between modalities, not a distribution over the state. With one modality missing, nothing tells the model how much state uncertainty remains. The SleepFM retrain lets us test whether cross-modal similarity from the remaining sensors happens to track error anyway. If it does, the generative decoder is unnecessary for this goal.

Why not just train one classifier per sensor subset? With four modality groups this is feasible. It is likely the most accurate option, which is why it serves as the upper bound. But it multiplies models by fifteen, produces no imputed signals, and gives each model only temperature-scaled confidence. We measure its calibration alongside ours rather than assuming it is worse.

Proposed Experiments

The primary benchmark is PhysioNet 2018 with EEG, EOG, EMG, and ECG, using the downstream tasks and subject splits of the masked-autoencoder paper. We evaluate every one of the fifteen missing patterns and report per-pattern AUROC, calibration error, entropy-error rank correlation, and risk-coverage curves.

To test the central-autonomic setting, where lags are long, we use two further datasets. StressID provides ECG, EDA, and respiration. DEAP provides EEG plus peripheral channels; there we compare against the stacked autoencoder of Multimodal deep learning approach for joint EEG-EMG data compression and classification on its reported 78.1% dominance and 65.9% arousal accuracy. We also report the learned delay kernels. If they are physiologically sensible, EDA should peak seconds after EEG; this is a sanity check, not a claim.

Imputation quality for missing waveforms is scored by continuous ranked probability score and by interval coverage against the autoencoder's point imputations.

The decisive check uses real rather than simulated failures. EPIStress records in-hospital EEG headbands and wristbands, which suffer motion artifacts. No reported predictive result was found on Lacuna for it. Without retraining, we test whether posterior width rises on segments that annotators or accelerometer thresholds flag as artifact-corrupted, and whether abstaining on high-entropy segments reduces error on its NASA-TLX-derived workload labels. A model whose widths respond only to sensors being absent, and not to sensors being present but corrupted, fails the claim as stated.