Towards navigation in endoscopic kidney surgery based on preoperative imaging

Ayberk Acar, Daiwei Lu, Yifan Wu, İpek Oğuz, Nicholas Kavoussi, Jie Ying Wu

TL;DR

This paper introduces a novel pipeline for intraoperative navigation in endoscopic kidney surgery. By registering 3D reconstructions from endoscopic videos with pre-operative CT scans, it creates a patient-specific 3D map that guides surgeons, aiming to reduce re-operation rates caused by incomplete exploration of the kidney's collecting system.

What Problem Does This Paper Address?

Endoscopic kidney surgery, while minimally invasive, faces a significant challenge: high re-operation rates. This is largely due to the limited field of view and depth perception of current endoscopes, making it difficult for surgeons to mentally map the complex, branched structure of the kidney's collecting system onto pre-operative CT scans. This "mental mapping" is crucial for identifying and treating conditions like kidney stones and tumors. When surgeons can't fully visualize and navigate the entire collecting system, residual stones or tumors can be missed, necessitating further procedures. The paper aims to bridge this gap by providing an objective, real-time navigational aid.

Methods

The proposed solution is a multi-stage pipeline:

  1. Automated Segmentation and 3D Modeling of the Kidney Collecting System: The process begins with segmenting the kidney's collecting system from pre-operative CT scans. This is achieved using a 3D U-Net model, a deep learning architecture well-suited for volumetric medical image segmentation. After initial segmentation, a post-processing step refines the extraction of the collecting system. The segmented data is then used to create a 3D mesh model of the kidney's internal structure.
  2. 3D Reconstruction from Endoscope Images: Simultaneously, endoscopic videos captured during surgery are processed to create a 3D reconstruction of the visible parts of the collecting system. The authors explore and compare different Structure from Motion (SfM) algorithms, including COLMAP, and a more advanced framework called Pixel-Perfect SfM, which leverages SuperPoint for feature detection and SuperGlue for feature matching. The goal is to obtain a dense point cloud representing the navigated anatomy.
  3. Endoscope Registration to CT: The crucial step is registering the 3D reconstruction from the endoscope to the pre-operative 3D CT model. This is accomplished using 3D registration techniques, specifically employing the Open3D library. The process involves voxelization, downsampling, normal estimation, and feature histogram calculation (FPFH). While RANSAC was initially considered for global registration, the authors found manual tuning of scales and reference points to be more stable for real-world, potentially deformed kidney anatomy. Iterative Closest Point (ICP) is then used for refinement to estimate the endoscope's pose (position and orientation) within the CT model for each frame.

Datasets

The study utilized a dataset of 17 delayed-phase CT scans from patients with various conditions, including healthy individuals, upper tract urothelial carcinoma, and kidney stones. These scans were manually labeled with the assistance of a surgeon. For evaluation, the authors also used real endoscopic videos and rendered videos from manually segmented CT scans.

Key Insights and Ideas

  • Bridging the Visual-CT Gap: The core idea is to create a direct link between the real-time visual feedback from an endoscope and the detailed anatomical map provided by pre-operative CT. This eliminates the need for a surgeon's potentially error-prone mental reconstruction.
  • SfM for Endoscopic Data: The paper highlights the challenges of applying standard SfM techniques to endoscopic kidney surgery due to factors like fluid motion and organic debris, which obscure features. They demonstrate that advanced feature detectors (SuperPoint) and matchers (SuperGlue), especially when refined (Pixel-Perfect SfM), yield more complete reconstructions than traditional methods like COLMAP.
  • Registration Robustness: Achieving accurate registration between the reconstructed endoscope data and the CT model is critical. The authors found that manual adjustments for scale and reference points were necessary for real-world data, suggesting that deformations in the kidney during surgery can impact automated registration.
  • 3D Map as a Guidance Tool: The output is a patient-specific 3D map that visualizes the collecting system, the reconstructed endoscope path, and the explored areas. This map, augmented with RGB endoscope images, serves as a powerful visual aid.

Key Results

  • Quantitative Registration Accuracy: The registration process achieved sub-millimeter errors, with an Inlier RMSE of 0.4335 mm for rendered videos and 0.7052 mm for real endoscopic videos.
  • Reconstruction Completeness: Methods using SuperPoint and SuperGlue generally provided more complete reconstructions of the collecting system compared to COLMAP, with Pixel-Perfect refinement further improving the structural outline for registration.
  • Qualitative Demonstration: The paper presents visual results (Figures 5 and 6) showing the successful registration of reconstructed endoscope paths onto the CT-derived kidney models, demonstrating good coverage of the navigated areas.
  • Runtime: The complete pipeline takes approximately 8 hours on a high-performance workstation, though this can vary based on system specifications and desired quality.

Limitations and Future Work

  • Real-time Processing: The current pipeline does not support real-time processing, which is a significant limitation for immediate intraoperative guidance.
  • Video Quality Dependence: Reconstruction accuracy is heavily influenced by the quality of the endoscopic video, with fluid motion and debris posing significant challenges.
  • Deformation Handling: While partial registration and dilation of the CT model are used to mitigate deformations, inexact matches and contractions can still occur.
  • Ground Truth for Evaluation: Evaluating registration accuracy is difficult without readily available ground truth for camera positions in real surgical scenarios.
  • Future Directions: The authors propose extending testing with realistic phantoms and ex-vivo studies for better ground truth collection. Future work will focus on enabling real-time processing, potentially through training feature extractors on kidney-specific landmarks, and exploring Simultaneous Localization and Mapping (SLAM) algorithms. They also aim to improve registration accuracy and automate scaling for a fully autonomous system.

Summary updated

Abstract

Endoscopic renal surgeries have high re-operation rates, particularly for lower volume surgeons. Due to the limited field and depth of view of current endoscopes, mentally mapping preoperative computed tomography (CT) images of patient anatomy to the surgical field is challenging. The inability to completely navigate the intrarenal collecting system leads to missed kidney stones and tumors, subsequently raising recurrence rates. A guidance system is proposed to estimate the endoscope positions within the CT to reduce re-operation rates. A Structure from Motion algorithm is used to reconstruct the kidney collecting system from the endoscope videos. In addition, the kidney collecting system is segmented from CT scans using 3D U-Net to create a 3D model. The two collecting system representations can then be registered to provide information on the relative endoscope position. Correct reconstruction and localization of intrarenal anatomy and endoscope position is demonstrated. Furthermore, a 3D map is created supported by the RGB endoscope images to reduce the burden of mental mapping during surgery. The proposed reconstruction pipeline has been validated for guidance. It can reduce the mental burden for surgeons and is a step towards the long-term goal of reducing re-operation rates in kidney stone surgery.
BibTeX
@inproceedings{acar2023towards,
  author = {Ayberk Acar and Daiwei Lu and Yifan Wu and İpek Oğuz and Nicholas Kavoussi and Jie Ying Wu},
  title = {{Towards navigation in endoscopic kidney surgery based on preoperative imaging}},
  booktitle = {Healthcare},
  year = {2023},
  doi = {10.1049/htl2.12059},
}
Figure