Stereo Dense Scene Reconstruction and Accurate Laparoscope Localization for Learning-Based Navigation in Robot-Assisted Surgery

Ruofeng Wei, Bin Li, Hangjie Mo, Bo Lu, Yonghao Long, Bohan Yang, Qi Dou, Yunhui Liu, Dong Sun

Abstract

This paper presents a learning-driven framework for dense 3D scene reconstruction and precise laparoscope localization using only stereo images. By integrating a fine-tuned deep stereo matching network with surfel-based reconstruction and a coarse-to-fine visual localization module, the system achieves sub-2mm reconstruction accuracy and robust camera tracking. This approach eliminates the need for bulky external tracking hardware, offering a streamlined solution for navigation in robot-assisted minimally invasive surgery.

Introduction

In Minimally Invasive Surgery (MIS), surgeons often operate with a limited 2D view of the surgical field, losing critical 3D depth perception. While modern navigation systems attempt to solve this by overlaying 3D anatomical models, they typically rely on external optical or electromagnetic sensors for camera tracking. These sensors add significant cost, complexity, and physical clutter to the operating room.

Estimating depth and camera pose directly from video (SLAM) is a viable alternative but remains challenging due to the nature of human tissue. Surgical environments are often texture-less, homogeneous in color, and highly specular, causing traditional feature-matching algorithms to fail. This research addresses these hurdles by combining deep learning for depth perception with robust geometric reconstruction techniques.

Methodology

The proposed pipeline consists of three primary modules designed to work in tandem using only stereo image pairs:

  1. Single-Frame Depth Estimation: The system utilizes a Hierarchical Deep Stereo Matching (HSM) network. To overcome the scarcity of surgical ground-truth data, the model is pretrained on autonomous driving data (KITTI) and fine-tuned on surgical images using unsupervised photometric loss and edge-aware smoothness.
  2. Dense Visual Reconstruction: Estimated depth maps are fused into a global 3D model using "surfels" (surface elements). This memory-efficient representation allows for real-time updates by minimizing geometric and photometric errors during frame-to-frame movement.
  3. Coarse-to-Fine Localization: To localize the camera within the reconstructed map, a two-step process is employed. First, a lightweight "student" network (trained via knowledge distillation from NetVLAD) performs coarse retrieval of historical frames. Second, fine localization is achieved by matching 2D pixels to 3D points using ORB features and the Perspective-n-Point (PnP) algorithm with RANSAC.
Overview of the proposed framework, detailing the 3D reconstruction pipeline (a), the depth estimation network (b), and the feature extraction module (c).

Datasets and Experimental Setup

The framework was validated across several diverse datasets to ensure robustness:

  • SCARED: Porcine data from a da Vinci Xi robot with structured light ground truth.
  • Ex-vivo Data: Custom stereo videos of uterus phantoms and tissues.
  • In-vivo Data: 95 high-quality video clips from robotic prostatectomy surgeries used for qualitative assessment.
  • Pretraining/Fine-tuning: SERV-CT, KITTI, and Google Landmarks datasets were used for various stages of model training and knowledge distillation.
Visual samples from the SCARED, Phantom, and in-vivo datasets illustrating the challenging surgical environments.

Results and Discussion

The system demonstrated high accuracy across all metrics:

  • Depth and Reconstruction: The fine-tuned HSM network achieved a Root Mean Square Error (RMSE) of 2.959 mm on the SCARED dataset. The final surfel-based reconstruction achieved an RMSE of under 1.71 mm, operating at approximately 12 frames per second (fps).
  • Localization: The coarse-to-fine module tracked the laparoscope with an average translation error of 2.17 mm and a rotation error of 2.23°, significantly outperforming baseline methods like MapNet.

Key insights include the effectiveness of autonomous driving datasets for pretraining surgical models and the utility of knowledge distillation for creating computationally efficient retrieval networks suitable for the operating room.

Conclusion

This framework proves that accurate, image-only surgical navigation is feasible by coupling deep-learning depth estimation with classical reconstruction backends. While currently limited by a static scene assumption and localization latency (~2.8 fps), future work will focus on handling tissue deformation and optimizing the retrieval network for faster performance, particularly in rigid environments like Functional Endoscopic Sinus Surgery (FESS).

Summary updated

BibTeX
@article{wei2023stereo,
  author = {Ruofeng Wei and Bin Li and Hangjie Mo and Bo Lu and Yonghao Long and Bohan Yang and Qi Dou and Yunhui Liu and Dong Sun},
  title = {{Stereo Dense Scene Reconstruction and Accurate Laparoscope Localization for Learning-Based Navigation in Robot-Assisted Surgery}},
  journal = {IEEE Trans. Biomedical Engineering},
  year = {2023},
  doi = {10.1109/tbme.2022.3195027},
}
Figure