Fusion-Vital: Video-RF Fusion Transformer for Advanced Remote Physiological Measurement

Jae-Ho Choi, Ki-Bong Kang, Kyung-Tae KimAAAI 2024 · DOI

Abstract

Remote physiological monitoring (rPPG) is essential for non-contact healthcare, but current single-sensor methods face significant challenges in uncontrolled environments. This paper introduces Fusion-Vital, a multimodal transformer-based architecture that fuses RGB video and Radio Frequency (RF) signals. By leveraging the complementary nature of these sensors—spatial resolution from cameras and depth sensitivity from RF—Fusion-Vital achieves robust vital sign estimation even in total darkness or under physical occlusion. Experimental results show substantial improvements in accuracy for both heart rate and respiration rate monitoring.

Introduction

Remote physiological monitoring aims to track vital signs like heart rate and respiration without physical contact. However, existing methods typically rely on a single modality, each with inherent "blind spots." RGB cameras are effective at detecting subtle skin color changes (blood volume pulse) but fail in low-light conditions or when the subject's face is obscured. Conversely, RF sensors excel at detecting chest wall micro-vibrations regardless of lighting but suffer from poor lateral resolution and sensitivity to lateral body movements.

Fusion-Vital addresses these challenges through a multimodal fusion strategy that effectively aligns 2D pixel data with complex radio reflections, creating a system capable of operating in diverse real-world scenarios.

Methodology

The Fusion-Vital architecture integrates three primary innovations to bridge the gap between disparate sensor modalities.

Overview of the Fusion-Vital architecture, showing the parallel video and RF encoding branches and the Fusion Transformer modules.

1. Time-Difference Alignment

To create a "common language" for fusion, both modalities are projected into a shared time-difference domain. Video data uses normalized frame differences, while RF signals are processed into Doppler frequency shifts via Short-Time Fourier Transform (STFT). This focus on motion helps isolate physiological signals from static background noise.

2. Parallel Encoders

The model employs specialized branches for each modality:

  • Video Branch: Utilizes 2D convolutions with Temporal Shift Modules (TSM) to capture temporal dynamics efficiently. A spatial soft-attention mask is applied to focus the model on the subject's face.
  • RF Branch: Processes Range-Joint-Time-Frequency (RJTF) maps through a 2D convolutional pipeline designed to mirror the video branch's structure, facilitating downstream fusion.

3. Fusion Transformer

The core of the integration is a Fusion Transformer using Multi-Head Cross-Attention (MCA). This mechanism allows the network to adaptively weight the most reliable sensor based on the environment. For instance, in dark settings, the model learns to prioritize RF features, while in cases of RF noise from lateral motion, it relies more heavily on video data.

Experiments and Datasets

The model was evaluated on two datasets:

  • RRM-static: A public respiration benchmark.
  • MMD-rPPG: A new dataset released by the authors containing 3 hours of synchronized video and RF data from 15 participants. This dataset specifically includes challenging scenarios like total darkness and physical occlusions to test system robustness.

Results and Discussion

Fusion-Vital significantly outperforms single-sensor baselines across all metrics. The physics-based complementarity of the sensors—where cameras provide lateral resolution and RF provides depth—allows the model to fill geometric sensing gaps.

Comparison of Fusion-Vital against state-of-the-art single-modality methods for respiration and heart rate estimation.

Key performance highlights include:

  • Respiration: An 83.4% reduction in Mean Absolute Error (MAE) compared to the best RGB-only model.
  • Heart Rate: A 33.2% reduction in MAE over RGB-only models and a 26.5% reduction over RF-only models.
  • Robustness: In "Dark" and "Occluded" scenarios where single-sensor models typically fail, Fusion-Vital maintains high accuracy by successfully switching attention to the functioning modality.

Conclusion

Fusion-Vital demonstrates the power of multimodal fusion for remote health monitoring. While the dual-branch architecture introduces some computational overhead, its ability to maintain accuracy in extreme conditions marks a significant step forward. Future work will focus on scaling the dataset to improve generalization across diverse populations and enhancing performance during vigorous physical activity.

Summary updated

BibTeX
@inproceedings{choi2024fusion,
  author = {Jae-Ho Choi and Ki-Bong Kang and Kyung-Tae Kim},
  title = {{Fusion-Vital: Video-RF Fusion Transformer for Advanced Remote Physiological Measurement}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i2.27898},
}
Figure