Abstract
This paper presents a learning-driven framework for dense 3D scene reconstruction and precise laparoscope localization using only stereo images. By integrating a fine-tuned deep stereo matching network with surfel-based reconstruction and a coarse-to-fine visual localization module, the system achieves sub-2mm reconstruction accuracy and robust camera tracking. This approach eliminates the need for bulky external tracking hardware, offering a streamlined solution for navigation in robot-assisted minimally invasive surgery.
Introduction
In Minimally Invasive Surgery (MIS), surgeons often operate with a limited 2D view of the surgical field, losing critical 3D depth perception. While modern navigation systems attempt to solve this by overlaying 3D anatomical models, they typically rely on external optical or electromagnetic sensors for camera tracking. These sensors add significant cost, complexity, and physical clutter to the operating room.
Estimating depth and camera pose directly from video (SLAM) is a viable alternative but remains challenging due to the nature of human tissue. Surgical environments are often texture-less, homogeneous in color, and highly specular, causing traditional feature-matching algorithms to fail. This research addresses these hurdles by combining deep learning for depth perception with robust geometric reconstruction techniques.
Methodology
The proposed pipeline consists of three primary modules designed to work in tandem using only stereo image pairs:
- Single-Frame Depth Estimation: The system utilizes a Hierarchical Deep Stereo Matching (HSM) network. To overcome the scarcity of surgical ground-truth data, the model is pretrained on autonomous driving data (KITTI) and fine-tuned on surgical images using unsupervised photometric loss and edge-aware smoothness.
- Dense Visual Reconstruction: Estimated depth maps are fused into a global 3D model using "surfels" (surface elements). This memory-efficient representation allows for real-time updates by minimizing geometric and photometric errors during frame-to-frame movement.
- Coarse-to-Fine Localization: To localize the camera within the reconstructed map, a two-step process is employed. First, a lightweight "student" network (trained via knowledge distillation from NetVLAD) performs coarse retrieval of historical frames. Second, fine localization is achieved by matching 2D pixels to 3D points using ORB features and the Perspective-n-Point (PnP) algorithm with RANSAC.
Datasets and Experimental Setup
The framework was validated across several diverse datasets to ensure robustness:
- SCARED: Porcine data from a da Vinci Xi robot with structured light ground truth.
- Ex-vivo Data: Custom stereo videos of uterus phantoms and tissues.
- In-vivo Data: 95 high-quality video clips from robotic prostatectomy surgeries used for qualitative assessment.
- Pretraining/Fine-tuning: SERV-CT, KITTI, and Google Landmarks datasets were used for various stages of model training and knowledge distillation.
Results and Discussion
The system demonstrated high accuracy across all metrics:
- Depth and Reconstruction: The fine-tuned HSM network achieved a Root Mean Square Error (RMSE) of 2.959 mm on the SCARED dataset. The final surfel-based reconstruction achieved an RMSE of under 1.71 mm, operating at approximately 12 frames per second (fps).
- Localization: The coarse-to-fine module tracked the laparoscope with an average translation error of 2.17 mm and a rotation error of 2.23°, significantly outperforming baseline methods like MapNet.
Key insights include the effectiveness of autonomous driving datasets for pretraining surgical models and the utility of knowledge distillation for creating computationally efficient retrieval networks suitable for the operating room.
Conclusion
This framework proves that accurate, image-only surgical navigation is feasible by coupling deep-learning depth estimation with classical reconstruction backends. While currently limited by a static scene assumption and localization latency (~2.8 fps), future work will focus on handling tissue deformation and optimizing the retrieval network for faster performance, particularly in rigid environments like Functional Endoscopic Sinus Surgery (FESS).
Summary updated