Segmentation of kidney stones in endoscopic video feeds

Zachary A. Stoebner, Daiwei Lu, Seok Hee Hong, Nicholas L. Kavoussi, Ipek Oguz

Abstract

Image segmentation has been increasingly applied in medical settings as recent developments have skyrocketed the potential applications of deep learning. Urology, specifically, is one field of medicine that is primed for the adoption of a real-time image segmentation system with the long-term aim of automating endoscopic stone treatment. In this project, we explored supervised deep learning models to annotate kidney stones in surgical endoscopic video feeds. In this paper, we describe how we built a dataset from the raw videos and how we developed a pipeline to automate as much of the process as possible. For the segmentation task, we adapted and analyzed three baseline deep learning models -- U-Net, U-Net++, and DenseNet -- to predict annotations on the frames of the endoscopic videos with the highest accuracy above 90%. To show clinical potential for real-time use, we also confirmed that our best trained model can accurately annotate new videos at 30 frames per second. Our results demonstrate that the proposed method justifies continued development and study of image segmentation to annotate ureteroscopic video feeds.

Introduction

In urology, ureteroscopy is a common minimally invasive procedure used to treat kidney stones. However, the surgical environment is challenging: the field of view is often obscured by blood, saline injection, or debris, making it difficult for surgeons to maintain a clear "lock" on the stone. While deep learning has revolutionized medical imaging, surgical specialties like urology have been slow to adopt these tools. The primary bottleneck is the lack of high-quality, annotated datasets. This research addresses two hurdles: the creation of a specialized dataset from raw surgical feeds and the implementation of a model fast enough to assist a surgeon in real-time.

Methods and Materials

The study evaluated three encoder-decoder architectures to find the optimal balance of accuracy and speed:

  1. U-Net: A standard for biomedical segmentation using skip connections to preserve spatial information.
  2. U-Net++: A variant using nested, dense skip pathways to bridge the semantic gap between encoder and decoder features.
  3. DenseNet (DenseNet67): An architecture utilizing dense blocks for feature reuse.

An automated preprocessing pipeline was developed using Otsu thresholding and OpenCV contour detection to isolate the circular endoscopic view from the rectangular frame. Transfer learning was employed, initializing models with ImageNet weights.

Dataset

A novel dataset was derived from 29 raw digital ureteroscopy videos. After quality assurance, 27 videos were used, yielding 676 frames for training/validation and 73 frames for a "challenging" test set. The test set included real-world difficulties like motion blur, debris fragmentation, and saline-induced cloudiness. Manual annotations were performed using polylines and converted into binary masks.

Results

Both U-Net and U-Net++ achieved validation Dice scores above 90%. On the challenging test set, U-Net++ reached a Dice score of 0.83 and an AUC of 0.98.

Comparison of training performance (Dice score and BCE loss) for U-Net, U-Net++, and DenseNet architectures.

The inference script processed video at 30 FPS on an NVIDIA RTX 2080 Ti, meeting the threshold for real-time clinical relevance. Qualitatively, the models successfully segmented stones even when partially obscured by laser lithotripsy "dust."

Discussion

A significant finding was the impact of pretraining. Initializing with ImageNet weights accelerated the training process by approximately 80%, leading to faster convergence and higher maximal accuracy compared to training from scratch.

Training curves comparing models trained from scratch (V) versus those using ImageNet pretraining (P), demonstrating significant gains in convergence speed and accuracy.

U-Net++ proved to be the most robust architecture, likely due to its sophisticated skip connections which handled surgical video noise better than standard U-Net or DenseNet. DenseNet performed significantly worse and was slower to train, suggesting that increased depth is not always beneficial for this specific domain.

Conclusion and Future Work

This work demonstrates the feasibility of real-time kidney stone segmentation. Future research will focus on expanding the dataset size, potentially using current models for "auto-annotation." We also aim to implement instance segmentation to distinguish individual fragments and incorporate temporal logic (e.g., self-attention) for smoother tracking across video frames. Finally, we plan to integrate this system directly with surgical hardware for live operating room overlays.

Summary updated

Abstract

Image segmentation has been increasingly applied in medical settings as recent developments have skyrocketed the potential applications of deep learning. Urology, specifically, is one field of medicine that is primed for the adoption of a real-time image segmentation system with the long-term aim of automating endoscopic stone treatment. In this project, we explored supervised deep learning models to annotate kidney stones in surgical endoscopic video feeds. In this paper, we describe how we built a dataset from the raw videos and how we developed a pipeline to automate as much of the process as possible. For the segmentation task, we adapted and analyzed three baseline deep learning models -- U-Net, U-Net++, and DenseNet -- to predict annotations on the frames of the endoscopic videos with the highest accuracy above 90\%. To show clinical potential for real-time use, we also confirmed that our best trained model can accurately annotate new videos at 30 frames per second. Our results demonstrate that the proposed method justifies continued development and study of image segmentation to annotate ureteroscopic video feeds.
BibTeX
@inproceedings{stoebner2022segmentation,
  author = {Zachary A. Stoebner and Daiwei Lu and Seok Hee Hong and Nicholas L. Kavoussi and Ipek Oguz},
  title = {{Segmentation of kidney stones in endoscopic video feeds}},
  booktitle = {Image Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=KV78mcmiKe},
}
Figure