Repetitive Motion Estimation Network: Recover cardiac and respiratory signal from thoracic imaging

Xiaoxiao Li, Vivek K. Singh, Yifan Wu, Klaus J. Kirchberg, James S. Duncan, Ankur KapoorarXiv 2018 · arXiv

Introduction

During cardiac interventions, clinicians frequently utilize projective X-ray fluoroscopy to visualize the beating heart. However, the concurrent motion of the heart and lungs introduces significant artifacts and misalignments between live anatomical structures and static guidance overlays. While external monitoring devices or implanted markers can mitigate these issues, they increase procedural complexity and invasiveness.

Extracting motion signals directly from the video feed is a preferred alternative, yet it presents substantial challenges. Cardiac X-rays often lack distinct, easily trackable Regions of Interest (ROIs), making the generation of dense, frame-by-frame annotations—typically required for training standard computer vision models—prohibitively difficult. The Repetitive Motion Estimation Network (RMEN) addresses this by recovering complex, overlapping motion signals using only sparse, easily obtained annotations.

Methodology

The RMEN framework employs a hybrid spatial-temporal architecture designed to map high-dimensional video data to a 1D motion manifold.

Pseudo-Target Generation

To circumvent the need for manual bounding box annotations, the authors utilize synchronized ECG QRS peaks as "phase boundaries." These peaks are assigned a value of 1, while midpoints between peaks are assigned -1. Linear interpolation followed by a sine transformation produces a smooth, continuous 1D target curve. This approach provides a dense supervisory signal from sparse temporal markers.

RMEN Architecture

The network processes cropped video sequences through a multi-stage pipeline:

  1. Spatial Encoding: Shared Conv2D blocks extract spatial features from individual frames.
  2. Temporal Encoding: A two-layer stacked ConvLSTM captures inter-frame temporal dynamics.
  3. Feature Fusion: A Conv3D layer integrates spatio-temporal features.
  4. Phase Regression: Fully connected layers regress these features into the final 1D phase curve.
The RMEN architecture consists of spatial feature encoding, spatial-temporal feature learning via ConvLSTM, and feature map fusion for 1D signal regression.

Signal Separation

The network's output represents a combined motion manifold. To isolate specific biological signals, classical frequency filtering is applied: a band-pass filter (0.5–2 Hz) for cardiac motion and a low-pass filter (< 0.33 Hz) for respiratory motion.

Experimental Results

The model was evaluated on two proprietary datasets of fluoroscopic videos (15 fps). Dataset A (629 videos) was used for training and validation, while Dataset B (329 videos) served as an independent test set to assess generalization.

Quantitative Performance

RMEN significantly outperformed baseline methods in peak detection accuracy. It achieved a mean peak detection offset of 0.88 frames, representing a 59.3% reduction in error compared to the best alternative model (PCA+LSTM). Furthermore, RMEN demonstrated superior robustness, producing only 27 false positives out of 1,703 peaks, whereas the unsupervised DensityFlow method produced 96.

Quantitative comparison of RMEN against baseline methods on Dataset B, highlighting superior offset accuracy and lower false positive rates.

Implicit Anatomical Tracking

A key insight from the study is that RMEN learns to identify relevant anatomical proxies without explicit guidance. Visualization of the Conv3D feature maps revealed that the network naturally focused on coronary arteries for cardiac motion and the diaphragm for respiratory motion. This suggests that the 1D manifold embedding effectively captures the underlying physical dynamics of the thoracic cavity.

Conclusion

The Repetitive Motion Estimation Network demonstrates that sparse temporal annotations are sufficient for tracking complex organ motion in medical imaging. By combining deep learning-based feature extraction with classical signal processing, RMEN provides a robust solution for motion compensation in image-guided surgeries. Future applications may include dynamic coronary roadmapping and real-time pacemaker tracking, offering a versatile blueprint for estimating repetitive motions in clinical environments.

Summary updated

BibTeX
@misc{li2018repetitive,
  author = {Xiaoxiao Li and Vivek K. Singh and Yifan Wu and Klaus J. Kirchberg and James S. Duncan and Ankur Kapoor},
  title = {{Repetitive Motion Estimation Network: Recover cardiac and respiratory signal from thoracic imaging}},
  year = {2018},
  eprint = {1811.03343},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.03343},
}
Figure