Segmentation of kidney stones in endoscopic video feeds

Zachary A. Stoebner, Daiwei Lu, Seok Hee Hong, Nicholas L. Kavoussi, Ipek Oguz

TL;DR

Researchers from Vanderbilt University have developed a real-time deep learning pipeline to automatically segment kidney stones in endoscopic video feeds. By adapting U-Net++ architectures and leveraging ImageNet pretraining, the system achieves over 90% accuracy and maintains a processing speed of 30 frames per second, paving the way for automated, image-guided urological surgery.

What Problem Does This Paper Address?

In urology, ureteroscopy is a common minimally invasive procedure used to treat kidney stones. However, the surgical environment is challenging: the field of view is often obscured by blood, saline injection, or debris, making it difficult for surgeons to maintain a clear "lock" on the stone.

While deep learning has revolutionized medical imaging, surgical specialties like urology have been slow to adopt these tools. The primary bottleneck isn't a lack of algorithms, but a lack of high-quality, annotated datasets. This paper tackles two hurdles: the creation of a specialized dataset from raw surgical feeds and the implementation of a model fast enough to assist a surgeon in real-time.

Methods

The authors evaluated three established encoder-decoder architectures to find the best balance of accuracy and speed:

  1. U-Net: The standard for biomedical segmentation, featuring skip connections to preserve spatial information.
  2. U-Net++: A more complex variant that uses nested, dense skip pathways to reduce the semantic gap between encoder and decoder features.
  3. DenseNet (specifically DenseNet67): An architecture that uses dense blocks to encourage feature reuse.

A key technical contribution is their automated preprocessing pipeline. Instead of manual cropping, they used Otsu thresholding and OpenCV contour detection to isolate the circular endoscopic view from the rectangular video frame. For training, they utilized transfer learning, initializing their models with weights from ImageNet—a technique that proved vital for performance.

Datasets

The study introduced a novel dataset derived from 29 raw digital ureteroscopy videos.

  • Final Count: After quality assurance, 27 videos were used.
  • Annotations: 676 frames were used for training/validation, and 73 frames were held out for a "challenging" test set.
  • Diversity: The test set was specifically curated to include "real-world" surgical difficulties, such as motion blur, debris fragmentation, and saline-induced cloudiness.
  • Labeling: Manual annotations were performed using polylines (via MakeSense.ai) and converted into binary masks.

Key Insights and Ideas

  • The Power of Pretraining: One of the most significant findings was that pretraining on ImageNet (a dataset of everyday objects) sped up the training process by approximately 80%. It allowed the models to converge much faster and reach higher maximal accuracies compared to training from scratch.
  • U-Net++ is the Sweet Spot: While U-Net is the industry standard, U-Net++ provided a measurable boost in robustness. It handled the "noise" of surgical video better than the other architectures, likely due to its more sophisticated skip connections.
  • DenseNet Struggles: Interestingly, DenseNet performed significantly worse and was much slower to train on a GPU, suggesting that "deeper" isn't always "better" for this specific type of medical video data.
  • Real-time is Ready: The authors demonstrated that the pipeline could process video at 30 FPS on an NVIDIA RTX 2080 Ti. This is a critical threshold for clinical relevance, as any lag would make the tool useless (or even dangerous) during a live procedure.

Key Results

  • Accuracy: Both U-Net and U-Net++ achieved validation accuracies (Dice scores) above 90%.
  • Test Performance: On the challenging hold-out test set, U-Net++ achieved a Dice score of 0.83 and an Area Under the Curve (AUC) of 0.98.
  • Speed: The inference script ran slightly faster than the duration of the videos themselves, confirming the feasibility of real-time surgical assistance.
  • Qualitative Success: The models successfully segmented stones even when they were partially obscured by debris or "dust" from laser lithotripsy.

Limitations and Future Work

Despite the promising results, the authors noted several areas for improvement:

  • Dataset Size: While 27 videos is a start, deep learning models thrive on thousands of examples. The authors hope to use their current model to "auto-annotate" new data to scale the dataset faster.
  • Granularity: The current model treats "clumps" of debris as a single stone. Future iterations aim to use instance segmentation to distinguish between individual fragments after a stone has been broken up by a laser.
  • Temporal Logic: The current model looks at frames in isolation. The authors plan to incorporate temporal models (like self-attention) to use information from previous frames, which would lead to smoother, more consistent tracking in video.
  • Hardware Integration: The next step is moving from post-processing recorded video to an "in-step" system that takes a direct feed from the surgical hardware (via DVI/HDMI) and overlays the segmentation on a monitor in the operating room.

Summary updated

Abstract

Image segmentation has been increasingly applied in medical settings as recent developments have skyrocketed the potential applications of deep learning. Urology, specifically, is one field of medicine that is primed for the adoption of a real-time image segmentation system with the long-term aim of automating endoscopic stone treatment. In this project, we explored supervised deep learning models to annotate kidney stones in surgical endoscopic video feeds. In this paper, we describe how we built a dataset from the raw videos and how we developed a pipeline to automate as much of the process as possible. For the segmentation task, we adapted and analyzed three baseline deep learning models -- U-Net, U-Net++, and DenseNet -- to predict annotations on the frames of the endoscopic videos with the highest accuracy above 90\%. To show clinical potential for real-time use, we also confirmed that our best trained model can accurately annotate new videos at 30 frames per second. Our results demonstrate that the proposed method justifies continued development and study of image segmentation to annotate ureteroscopic video feeds.
BibTeX
@inproceedings{stoebner2022segmentation,
  author = {Zachary A. Stoebner and Daiwei Lu and Seok Hee Hong and Nicholas L. Kavoussi and Ipek Oguz},
  title = {{Segmentation of kidney stones in endoscopic video feeds}},
  booktitle = {Image Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=KV78mcmiKe},
}

Loading related content...

Figure