Learning Higher-Order Dynamics in Video-Based Cardiac Measurement

Brian L. Hill, Xin Liu, Daniel McDuff

TL;DR

Most video-based cardiac monitoring focuses on first-order changes to extract coarse metrics like heart rate, missing subtle waveform dynamics essential for diagnosing cardiovascular disease. By feeding neural networks second-derivative inputs (difference-of-difference frames) and supervising them with acceleration-based loss functions, this paper significantly improves the estimation of delicate cardiac timing features like Left Ventricular Ejection Time (LVET).

What Problem Does This Paper Address?

Remote photoplethysmography (iPPG) extracts cardiovascular pulse signals by measuring imperceptible skin color fluctuations from standard video. While existing deep learning methods can reliably estimate overall heart rate (HR), they struggle to capture accurate waveform morphology.

Clinically critical physiological events—such as the dicrotic notch (caused by the closure of the aortic valve) and the systolic foot—often appear as mere subtle inflections in raw pulse signals, making them difficult to detect. In the pulse's second derivative (acceleration PPG), however, these inflections become distinct local maxima. These higher-order dynamics are vital for calculating Left Ventricular Ejection Time (LVET), an important indicator of vascular aging, arterial stiffness, and heart failure risk. Standard video models optimize only for zeroth- or first-order dynamics (e.g., optical flow, inter-frame differences), sacrificing the high-frequency temporal fidelity needed to recover these morphological landmarks.

Methods

The authors build on Shafer’s Dichromatic Reflection Model (DRM) to theoretically demonstrate that second-order temporal derivatives of pixel intensities are dominated by the second derivative of the pulsatile blood volume. Guided by this, they propose a multi-derivative architecture:

  • Three-Branch Convolutional Attention Network (CAN):
  • Appearance Branch: Consumes raw frames to learn a shared spatial soft-attention mask, isolating facial skin and suppressing background noise.
  • First-Derivative Branch: Processes normalized difference frames ($X'(t) = X(t) - X(t-1)$).
  • Second-Derivative Branch: Processes difference-of-difference frames ($X''(t) = X'(t) - X'(t-1)$).
  • Temporal Modeling: Extracted spatial features from the derivative branches are concatenated and passed to bidirectional GRU layers followed by standard GRU layers.
  • Higher-Order Loss Supervision: The network is trained with an explicit mean squared error (MSE) loss on the acceleration PPG waveform ($p''(t)$) rather than relying solely on velocity ($p'(t)$).
Raw Video Frames ──────────► [Appearance Branch] ──► Attention Mask
                                                          │ (shared)
Frame Differences (FD) ────► [FD Conv Branch] ───────────(x)──┐
                                                              ├──► [RNN / GRU] ──► Target Derivatives
Difference-of-Differences ─► [SD Conv Branch] ───────────(x)──┘

Datasets

  • Training Data (Synthetic): 2,800 6-second synthetic avatar videos generated across varied skin tones, head poses, and lighting conditions. Pulse variations were driven using real clinical PPG waveforms sampled from the MIMIC PhysioNet database.
  • Evaluation Data (Real-world):
  • AFRL Dataset: 300 videos from 25 participants across controlled head motion tasks, benchmarked against contact finger PPG ground truth.
  • UBFC-Phys & PURE Datasets: Used in cross-dataset evaluations to verify out-of-domain generalization.

Key Insights and Ideas

  • Inflections Become Peaks: In raw or velocity-based PPG, closing of the aortic valve produces a faint plateau or inflection. Differentiating twice transforms this feature into a distinct mathematical maximum, easing temporal localization.
  • The Accuracy Trade-off: There is a clear Pareto trade-off between lower-order summary statistics and higher-order morphological precision. Models optimized solely on first-derivative signals achieve superior heart rate accuracy, but fail dramatically on fine timing intervals. Conversely, second-derivative optimization dramatically tightens morphology at a slight expense to global rate estimation.
  • Sim-to-Real Generalization: Models trained purely on synthetic avatars transferred effectively to real human subjects, demonstrating that simulated physics and varied avatar rendering can alleviate the lack of diverse, clinically annotated iPPG datasets.

Key Results

On the AFRL benchmark:

  • LVET Error Cut in Half: The Second-Derivative (SD)-Optimized model achieved a 52.07 ± 19.53 ms Mean Absolute Error (MAE) for LVET, outperforming the baseline First-Derivative (FD)-Optimized model (108.26 ± 56.19 ms).
  • The Frequency Trade-off: The baseline FD model attained the best overall HR MAE (0.66 ± 2.07 BPM vs. 3.41 ± 8.77 BPM for the SD model), illustrating the tension between optimizing for dominant frequency versus sharp waveform morphology.
  • Cross-Dataset Robustness: On external benchmarks, the SD model maintained roughly 50% lower LVET errors:
  • UBFC: 41.90 ms (SD) vs. 80.90 ms (FD)
  • PURE: 37.85 ms (SD) vs. 75.84 ms (FD)

Limitations and Future Work

  • Motion Sensitivity: Evaluations focused primarily on stationary and minimal-motion scenarios (AFRL tasks 1 and 2). Evaluating higher-order dynamics under vigorous motion remains an open challenge because acceleration terms amplify high-frequency noise.
  • Joint Optimization Balance: A combined objective incorporating both first- and second-derivative targets yielded intermediate results (LVET MAE of 65.99 ms; HR MAE of 1.16 BPM), showing that further architectural or loss-weighting work is needed to simultaneously master both summary rates and fine morphology.

Summary updated

Abstract

Computer vision methods typically optimize for first-order dynamics (e.g., optical flow). However, in many cases the properties of interest are subtle variations in higher-order changes, such as acceleration. This is true in the cardiac pulse, where the second derivative can be used as an indicator of blood pressure and arterial disease. Recent developments in camera-based vital sign measurement have shown that cardiac measurements can be recovered with impressive accuracy from videos; however, the majority of research has focused on extracting summary statistics such as heart rate. Less emphasis has been put on the accuracy of waveform morphology that is necessary for many clinically impactful scenarios. In this work, we provide evidence that higher-order dynamics are better estimated by neural models when explicitly optimized for in the loss function. Furthermore, adding second-derivative inputs also improves performance when estimating second-order dynamics. By incorporating the second derivative of both the input frames and the target vital sign signals into the training procedure, our model is better able to estimate left ventricle ejection time intervals.
BibTeX
@inproceedings{hill2022learning,
  author = {Brian L. Hill and Xin Liu and Daniel McDuff},
  title = {{Learning Higher-Order Dynamics in Video-Based Cardiac Measurement}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=xOeWOPFXrTh},
}
Figure