Multimodal Respiratory Rate Estimation From Audio and Video in Emergency Department Patients

John Harvill, Moitreya Chatterjee, Shaveta Khosla, Mustafa Alam, Narendra Ahuja, Mark Hasegawa‐Johnson, David Chestek, David G. BeiserIEEE Journal of Translational Engineering in Health and Medicine 2024 · DOI

TL;DR

This paper introduces a multimodal approach to estimate respiratory rate (RR) from audio and video signals of emergency department patients, aiming for contactless and remote vital sign monitoring. They found that video-based estimation generally outperforms audio, and while their neural network approach shows promise, signal processing methods using spectrograms were more effective on their dataset.

What Problem Does This Paper Address?

The COVID-19 pandemic highlighted the critical need for reliable, remote healthcare solutions, including telemedicine. A key challenge in virtual patient visits is the accurate assessment of vital signs, particularly respiratory rate (RR), which is a crucial indicator of patient health. Traditional RR measurement involves manual counting, which can be time-consuming and prone to error, especially in busy clinical settings or when dealing with factors like poor lighting or camera angles in telemedicine. This paper addresses the need for more efficient, accurate, and contactless methods for RR estimation, specifically exploring the potential of using readily available audio and video from patient smartphones or other devices.

Methods

The researchers explored two primary paradigms for RR estimation:

  • Autocorrelation: This signal processing-based approach looks for repetitive patterns in the feature domain (audio spectrograms or video features) to identify the underlying period of respiration. They extracted features from video by dividing frames into grids and computing mean intensity within each grid cell, and used log Mel spectrograms for audio. The most periodic features were selected based on variance, and then an autocorrelation function was applied.
  • Breathing Boundaries: This is a learning-based approach that uses a neural network to predict the boundaries between breaths. They trained a Bi-directional Long Short-Term Memory (BLSTM) network on a publicly available audio dataset (ICBHI) to classify frames as either breathing boundaries or non-boundaries. This approach was applied to audio only due to the lack of a large, annotated video corpus for training. They then used two methods to derive RR from these predicted boundaries: autocorrelation on the probability outputs and a Hidden Semi-Markov Model (HSMM).

A key aspect of their work is the multimodal fusion, where they combined features from both audio and video to potentially improve robustness.

Datasets

  • Custom Collected Dataset: Audio and video recordings were collected from 19 patients presenting to emergency departments at the University of Illinois at Chicago and the University of Chicago. These patients had respiratory complaints and were asked to perform various standardized tasks. This dataset was used for evaluating all proposed methods.
  • ICBHI Breathing Audio Dataset: This publicly available dataset was used for training the neural network in the "Breathing Boundaries" approach.

Key Insights and Ideas

  • Video's Stability: Video-based RR estimation proved to be more stable and less prone to outliers compared to audio-based methods on their collected data. This suggests that visual cues of chest movement might be more robust in certain clinical environments.
  • Spectrograms vs. Neural Predictions: For audio-only analysis, the signal processing approach using spectrograms (autocorrelation) outperformed their neural network-based "Breathing Boundaries" method. This indicates that for their dataset size and configuration, the raw periodicity information in the spectrogram was more informative than the learned boundary predictions.
  • Multimodal Fusion Potential: While their experiments showed video outperforming audio, the fusion of both modalities didn't consistently yield superior results over the best single modality. However, the authors suggest that with more sophisticated fusion strategies, multimodal approaches could overcome individual modality weaknesses.
  • Noise Robustness: Both the autocorrelation and Breathing Boundaries methods demonstrated good robustness to noise, with performance degradation only becoming significant at SNR levels of 5 dB or lower.

Key Results

  • Custom Dataset:
    • The best Mean Absolute Error (MAE) for video-based autocorrelation was 2.53.
  • The best MAE for audio-based autocorrelation was 3.67.
  • Fusion of audio and video features using autocorrelation achieved an MAE of 2.53.
  • The "Breathing Boundaries" approach achieved an MAE of 3.52, and the "Breathing Boundaries HSMM" achieved an MAE of 3.34.
  • ICBHI Dataset (for Breathing Boundaries training/testing):
    • The autocorrelation approach achieved an MAE of 1.63.
  • Outliers: Video-based estimation had fewer significant outliers than audio-based estimation on their dataset.
  • Higher RR Estimation: The Breathing Boundaries HSMM approach showed a slight improvement in accurately predicting higher respiratory rates compared to the standard Breathing Boundaries method.

Limitations and Future Work

  • Dataset Size: The custom dataset of 19 patients is relatively small, which may have limited the performance of the neural network approach. The authors suggest that more training data could potentially improve the neural network's performance.
  • Fusion Strategy: The multimodal fusion strategy was basic (combining features). Future work could explore more advanced fusion techniques.
  • Video Feature Extraction: The video feature extraction method was computationally inexpensive. More complex video analysis might yield better results but could increase computational load.
  • Telemedicine Setting Nuances: While the study was conducted in an ED setting, the authors envision future work focusing on improving techniques for the video modality, particularly for controlled telemedicine scenarios where patient positioning and lighting might be more consistent. They also suggest exploring methods to intelligently select the best modality (audio or video) for a given situation.

Summary updated

Abstract

Accurate, contactless estimation of vital signs using video or audio is significant, because it can be performed remotely. Additionally, it is contactless and does not require extra measurement equipment.
BibTeX
@article{harvill2024multimodal,
  author = {John Harvill and Moitreya Chatterjee and Shaveta Khosla and Mustafa Alam and Narendra Ahuja and Mark Hasegawa‐Johnson and David Chestek and David G. Beiser},
  title = {{Multimodal Respiratory Rate Estimation From Audio and Video in Emergency Department Patients}},
  journal = {IEEE Journal of Translational Engineering in Health and Medicine},
  year = {2024},
  doi = {10.1109/jtehm.2024.3418345},
}
Figure