ViTaL-Diff: Video-Token Latent Diffusion for Contactless Respiratory Monitoring

Shadman Sakib, Nirmalya RoySD4H @ ICML 2026 · OpenReview

TL;DR

ViTaL-Diff introduces a video-conditioned latent diffusion framework to reconstruct respiratory waveforms from standard video without contact sensors. By modeling breathing as a conditional distribution rather than a single deterministic value, the system achieves state-of-the-art accuracy and provides a built-in "reliability signal" that flags when visual evidence is too weak for a trustworthy reading.

What Problem Does This Paper Address?

Monitoring respiratory rate (RR) is critical for everything from neonatal care to sleep studies, but traditional contact sensors (like chest belts or wearables) are often uncomfortable and prone to displacement. Video-based monitoring is a promising contactless alternative, but it faces a major hurdle: breathing is a subtle signal.

Small chest movements are easily "drowned out" by changes in lighting, loose clothing, or non-respiratory body movements. Most existing deep learning models treat this as a deterministic problem—mapping a video clip to a single RR value. However, this is an ill-posed inverse problem; a single blurry video clip could plausibly represent several different breathing patterns. Deterministic models struggle here because they try to force a single "average" answer, which often leads to high error in ambiguous conditions.

Methods

The authors propose ViTaL-Diff, which shifts the task from simple regression to conditional generative modeling. The architecture consists of three main components:

  1. Respiratory Waveform Tokenizer: Instead of operating on raw, noisy 1D signals, the authors first train an autoencoder to learn a compact "latent space" of respiratory waveforms. This compresses the signal into discrete tokens that capture the essential structure of human breathing.
  2. Video Evidence Encoder: The input video is divided into spatiotemporal "tubelets" (small 3D chunks of video). These are embedded as visual tokens that provide the context for the diffusion process.
  3. Latent Diffusion Transformer: This is the core engine. During inference, the model starts with pure noise in the latent space and iteratively "denoises" it, guided by the video tokens. It uses cross-attention to align specific visual cues (like a shoulder rising) with the corresponding parts of the generated respiratory waveform.

By sampling from this model multiple times, the system produces a range of plausible waveforms. The average of these samples is the final estimate, while the variance between them serves as a measure of uncertainty.

Datasets

The model was evaluated across three diverse datasets to ensure it wasn't just "overfitting" to one specific camera setup:

  • In-house RGB Dataset: 50 videos of 10 volunteers in various postures (front-facing, supine, etc.) with synchronized ground-truth data from a Vernier respiration belt.
  • AIR-125: A public dataset containing 125 pediatric videos from real-world settings with varying resolutions and frame rates.
  • Sleep Dataset: 28 recordings of adults in low-light conditions using Infrared (IR) and Near-Infrared (NIR) video, where motion is extremely subtle.

Key Insights and Ideas

  • Latent Space is Key: The authors found that performing diffusion in a learned latent space is significantly more effective than diffusing in the raw waveform space. This is because the latent space constrains the model to only generate "physiologically plausible" breathing patterns, preventing the model from producing erratic, non-human signals.
  • Embracing Ambiguity: By using a generative approach, the model doesn't have to "guess" a single answer when the video is dark or the patient is moving. It produces a distribution. If the samples vary wildly, the system knows it's guessing and can signal that the data is unreliable.
  • Zero-Cost Uncertainty: Unlike other models that require a separate "confidence head" or auxiliary training, ViTaL-Diff provides uncertainty for free. The variance between generated samples naturally correlates with prediction difficulty.

Key Results

ViTaL-Diff outperformed classical signal processing (FFT), deterministic deep learning (RespFormer), and raw waveform diffusion models across the board:

  • Accuracy: It achieved a 35.4% reduction in Mean Absolute Error (MAE) compared to classical methods and up to a 28.9% reduction over the best deterministic deep learning baselines.
  • Aggregate Performance: Across all three datasets, it maintained a low average error of 3.47 breaths per minute (bpm).
  • Reliability: In an "uncertainty-coverage" analysis, the authors showed that by rejecting the top 30% of clips where the model was most "uncertain," the MAE dropped from 2.7 bpm to just 1.8 bpm. This proves the uncertainty metric is a highly effective filter for real-world deployment.

Limitations and Future Work

While ViTaL-Diff is a significant step forward, the authors acknowledge several areas for growth:

  • Total Occlusion: The model still relies on some visible motion. If a patient is completely under a heavy, opaque blanket, the visual evidence disappears entirely.
  • Computational Load: Diffusion models are iteratively sampled, which is more computationally expensive than a single "forward pass" of a deterministic model. This could pose challenges for real-time monitoring on low-power edge devices.
  • Clinical Validation: While tested on pediatric and sleep datasets, broader clinical trials are needed to see how the model handles patients with specific respiratory pathologies (like COPD or asthma) where breathing patterns are irregular.

Summary updated

Abstract

Contactless respiratory monitoring from video is a challenging inverse problem because breathing is observed indirectly through subtle body and clothing motion, which can be corrupted by illumination changes, occlusion, pose variation, and non-respiratory movement. We propose \textbf{ViTaL-Diff}, a video-token latent diffusion framework that reconstructs respiratory waveforms from short upper-body videos without contact sensors at inference. ViTaL-Diff learns a compact latent space from belt-derived respiratory waveforms and trains a video-conditioned diffusion transformer to generate respiratory latent tokens from spatiotemporal visual evidence. By modeling a distribution over plausible respiratory waveforms, ViTaL-Diff supports both respiratory-rate estimation and uncertainty quantification. Across three datasets, including an in-house RGB dataset, AIR-125, and a low-light Sleep dataset, ViTaL-Diff achieves the lowest error among classical, deterministic, and diffusion baselines, with up to 28.9\% mean absolute error (MAE) reduction over deterministic deep baselines. Its uncertainty estimates further identify visually ambiguous clips, providing practical reliability signal for contactless monitoring.
BibTeX
@inproceedings{sakib2026vital,
  author = {Shadman Sakib and Nirmalya Roy},
  title = {{ViTaL-Diff: Video-Token Latent Diffusion for Contactless Respiratory Monitoring}},
  booktitle = {SD4H @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=xtmIBacWMr},
}
Figure