Learning Higher-Order Dynamics in Video-Based Cardiac Measurement

Brian L. Hill, Xin Liu, Daniel McDuff

TL;DR

This paper demonstrates that video-based cardiac monitoring can move beyond simple heart rate estimation to capture clinically rich waveform morphology. By explicitly incorporating second-order dynamics (acceleration) into both the neural network inputs and the loss functions, the authors significantly improve the estimation of the Left Ventricle Ejection Time (LVET), a key indicator of cardiovascular health.

What Problem Does This Paper Address?

Remote photoplethysmography (rPPG) has become highly accurate at estimating heart rate (HR) from facial videos. However, HR is just a summary statistic. Much of the clinically relevant information—such as blood pressure, arterial stiffness, and vascular aging—is hidden in the waveform morphology (the specific shape of the pulse).

A major challenge is that key features, like the dicrotic notch (which signals the closing of the aortic valve), are often just subtle inflections in a raw PPG signal that are easily lost to noise or video compression. While these features are hard to see in the "position" (raw signal) or "velocity" (first derivative), they appear as distinct peaks in the "acceleration" (second derivative). Current computer vision models typically optimize for first-order changes (like optical flow), leaving a gap in our ability to recover these higher-order clinical markers.

Methods

The authors propose a multi-derivative approach that treats acceleration as a first-class citizen:

  1. Multi-Derivative Inputs: Instead of just using raw frames or normalized difference frames ($X'$), they introduce difference-of-difference frames ($X''$). This explicitly provides the model with pixel-level acceleration data.
  2. Higher-Order Loss Functions: They move beyond standard MSE on the raw signal. The models are trained with an explicit objective to minimize the error between the predicted and ground-truth second derivative of the PPG signal.
  3. Architectures: They adapted two existing state-of-the-art architectures:
  4. RhythmNet: A CNN + RNN pipeline.
  5. CAN (Convolutional Attention Network): A model that uses an attention branch to mask out non-skin pixels, which the authors modified to share attention masks across first- and second-derivative branches.
  6. Synthetic Training: To overcome the lack of diverse, labeled clinical data, they trained their models using synthetic avatars. These avatars were driven by real physiological waveforms from the MIMIC database, allowing for a diverse range of skin tones and pulse shapes.

Datasets

  • Training: A custom synthetic dataset of 2,800 6-second videos featuring highly parameterized avatars.
  • Evaluation: Three benchmark datasets of real human subjects:
  • AFRL: 300 videos from 25 participants with contact-sensor ground truth.
  • UBFC-RPPG: A standard public benchmark for remote heart rate.
  • PURE: A dataset focused on pulse rate estimation under motion.

Key Insights and Ideas

  • The "Acceleration" Advantage: The second derivative of a PPG signal (SDPPG) is much more sensitive to vascular properties than the raw signal. The authors show that if you want to measure the heart's timing (like LVET), you must look at the acceleration.
  • The Optimization Trade-off: There is a clear "no free lunch" result. Models optimized for first-order dynamics (FD-Optimized) are the champions of Heart Rate accuracy but fail miserably at capturing morphology. Conversely, SD-Optimized models are significantly better at clinical timing (LVET) but show slightly higher errors in basic heart rate.
  • Feature Preservation: Explicitly calculating the second derivative in the loss function forces the neural network to learn feature representations that preserve subtle inflections rather than smoothing them out as noise.

Key Results

  • LVET Accuracy: On the AFRL dataset, the SD-Optimized CAN model achieved a Mean Absolute Error (MAE) of 52.07 ms, compared to 108.26 ms for the standard FD-Optimized version—a massive improvement in clinical utility.
  • Heart Rate Performance: The FD-Optimized CAN model achieved a very low HR MAE of 0.66 BPM, while the SD-Optimized version rose to 3.41 BPM, confirming the trade-off between frequency accuracy and morphological precision.
  • Generalization: Models trained entirely on synthetic data generalized well to real-world datasets (UBFC and PURE), proving that synthetic avatars are a viable path for training specialized medical CV models.

Limitations and Future Work

  • Motion Sensitivity: The study focused on tasks with minimal head motion. Because acceleration is highly sensitive to movement, the authors acknowledge that large, sudden motions remain a significant hurdle for recovering subtle second-derivative signals.
  • Synthetic Realism: While synthetic data worked well, the authors suggest that increasing the diversity and realism of the avatars (e.g., better subsurface scattering models for skin) could further close the sim-to-real gap.
  • Clinical Validation: Future work could involve testing these models on populations with known cardiovascular diseases to see if the recovered LVET intervals can successfully categorize patients by arterial stiffness or heart failure risk.

Summary updated

Abstract

Computer vision methods typically optimize for first-order dynamics (e.g., optical flow). However, in many cases the properties of interest are subtle variations in higher-order changes, such as acceleration. This is true in the cardiac pulse, where the second derivative can be used as an indicator of blood pressure and arterial disease. Recent developments in camera-based vital sign measurement have shown that cardiac measurements can be recovered with impressive accuracy from videos; however, most of the research has focused on extracting summary statistics such as heart rate. Less emphasis has been put on the accuracy of waveform morphology that is necessary for many clinically meaningful assessments. In this work, we provide evidence that higher-order dynamics are better estimated by neural models when explicitly optimized for in the loss function. Furthermore, adding second-derivative inputs also improves performance when estimating second-order dynamics. We illustrate this, by showing that incorporating the second derivative of both the input frames and the target vital sign signals into the training procedure, models are better able to estimate left ventricle ejection time (LVET) intervals.
BibTeX
@inproceedings{hill2022learning,
  author = {Brian L. Hill and Xin Liu and Daniel McDuff},
  title = {{Learning Higher-Order Dynamics in Video-Based Cardiac Measurement}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=xOeWOPFXrTh},
}
Figure