Introduction
Minimally Invasive Cardiac Surgery (MICS) offers patients faster recovery and less trauma compared to open-heart surgery. However, surgeons operating through small incisions face a "keyhole" problem: they have a very narrow field of view, restricted movement, and a lack of 3D depth perception. While 3D reconstruction from video (Structure from Motion) is a well-established field, it usually assumes the object being filmed is rigid. The heart is the opposite of rigid—it pulsates and deforms constantly. This paper introduces a novel method for reconstructing the 3D surface of a beating heart using video from a moving endoscopic camera by combining heart-cycle synchronization with non-rigid Structure from Motion (NRSFM).
Methodology
The authors propose a multi-stage pipeline to handle the complexity of "4D" (3D + time) reconstruction:
- Pre-processing & Calibration: Zhang’s planar calibration is used to correct for barrel distortion in wide-angle endoscopic lenses.
- Heart Cycle Synchronization: To simplify the mathematics of deformation, the system identifies the heart's phase and extracts images captured at the same point in each cycle (usually end-diastole). This turns a chaotic deformation problem into a "residual motion" problem that is much easier for NRSFM algorithms to solve.
- Non-rigid Structure from Motion (NRSFM): A probabilistic model (Probabilistic PCA) and an Expectation-Maximization (EM) algorithm allow the recovery of 3D structure and camera motion simultaneously, even with missing data points.
- Iterative Outlier Removal: Instead of standard rigid constraints, the authors iteratively remove points with high "reprojection errors"—the difference between where a 3D point should appear on the 2D screen versus where it was actually tracked.
Experimental Validation
The method was validated using three distinct data sources:
- Synthetic Data: 100 points from a beating heart model were used to test robustness against noise and missing data.
- Heart Phantom: A silicone rubber heart phantom inflated/deflated by an air pump. Ground truth was established using a 64-slice CT scanner, with video captured via a da Vinci surgical system.
- In vivo Data: Real endoscopic footage from a Totally Endoscopic Coronary Artery Bypass (TECAB) procedure, with ECG signals providing ground truth for heart rate.
Key Results and Insights
- Accuracy: When compared against the CT scan of the phantom, the reconstructed 3D point cloud had a mean residual error of approximately 3.7 mm. In real surgical footage, the iterative outlier removal successfully reduced the average RMS error from 2.9 mm to 2.4 mm.
- Camera Mobility: Unlike previous stereo-based methods that required a stationary camera, this approach allows the endoscope to move around the scene, which is crucial for creating a wide field-of-view map of the surgical site.
- Robustness: Synthetic tests showed stability even when 20% of feature points were missing, a common occurrence in surgery due to camera movement or blood obscuring the view.
Limitations and Future Work
The primary limitation is computation time; the process currently takes several minutes, making it unsuitable for real-time surgical guidance. Future work aims to utilize Graphical Processing Units (GPUs) for acceleration and incorporate more sophisticated bio-mechanical models of the heart to improve accuracy. The ultimate goal is to automatically register these 3D surface reconstructions with preoperative CT or MRI scans to provide surgeons with enhanced guidance during procedures.
Summary updated