Multi-Stage Suture Detection for Robot Assisted Anastomosis Based on Deep Learning

Yang Hu, Yun Gu, Jie Yang, Guang-Zhong Yang

TL;DR

This paper introduces a novel multi-stage deep learning framework for accurately detecting surgical suture threads, even in challenging scenarios like self-occlusion. By combining a two-branched convolutional neural network for initial detection with a curvilinear structure detector, the method reconstructs the thread's path, paving the way for more robust robotic-assisted surgery.

What Problem Does This Paper Address?

Automated surgical procedures, particularly those involving suturing, are a major goal in robotic surgery. A critical component of this automation is the ability to accurately detect and track the suture thread. However, this task is fraught with difficulties: suture threads are thin, deformable, and prone to self-occlusion (like in knots) and occlusion by surgical tools or tissues. Existing methods often struggle with these complexities, either due to low-resolution sensing, reliance on complex physical models that fail with entanglement, or an inability to handle segmentation outliers. This paper aims to bridge this gap by proposing a robust, model-free approach for suture thread detection that can handle these challenging surgical conditions.

Methods

The proposed method, termed Multi-Stage Suture Detection (DMSD), employs a two-stage deep learning approach.

  1. Initial Detection (Two-Branched CNN): The first stage uses a two-branched fully convolutional neural network (CNN) to simultaneously predict two crucial pieces of information from an input image:
  2. Gradient Map: This map indicates the direction and intensity of the thread's centerline.
  3. Overlap Map: This map identifies regions where the thread might be self-occluding. These initial predictions are then fused with the original image.

  4. Refined Detection and Reconstruction: The fused information is fed into a second-stage CNN to generate a refined gradient map. This refined map is then processed by a curvilinear structure detector. This detector:

  5. Extracts thin centerline segments from the gradient map.
  6. Identifies potential endpoints of the thread using a polarity map.
  7. Uses a region-growing approach to link these segments into complete curve segments.
  8. Orders these segments based on their average intensity and fits a spline to reconstruct the final thread path.

A key aspect of this method is its "bottom-up" approach, focusing on extracting local features and then assembling them, rather than relying on a pre-defined "top-down" physical model of the thread.

Datasets

The authors collected their own dataset for training and evaluation. This dataset consists of:

  • Knot-tying procedures: Recorded videos of surgical knot-tying tasks.
  • Two types of threads:
    • Threads with directional patterns (to enhance visibility and potentially aid in distinguishing thread ends).
  • Single-color threads.
  • Ground Truth Annotation: Manual annotation involved marking key points along the thread centerline, indicating occlusion status, and generating gradient and overlap maps.

The paper does not mention using publicly available benchmarks for this specific task.

Key Insights and Ideas

  • The Power of Multi-Stage Fusion: Combining initial gradient and overlap predictions with the raw image in a second stage significantly improves the robustness of the thread detection, especially in challenging regions.
  • Overlap Map for Self-Occlusion: Explicitly predicting thread overlap is crucial for handling complex scenarios like knots, where traditional methods often fail.
  • Model-Free Reconstruction: Avoiding reliance on complex physical thread models makes the system more adaptable to various thread behaviors and less prone to failure when entanglement occurs.
  • Directional Markers Aid Detection: Threads with directional patterns demonstrably improve detection accuracy, particularly in difficult occlusion scenarios, highlighting the benefit of visual cues for automated systems.

Key Results

  • Quantitative Evaluation: The proposed DMSD method achieved higher accuracy than a baseline method (DMSD-S) in terms of Peak-Signal-to-Noise Ratio (PSNR) and Overall Thread Tracking Precision (OTTP).
  • For threads with markers, DMSD showed improved stability (smaller standard deviation).
  • For threads without markers, the improvement was significant, especially in OTTP, indicating better overall thread path reconstruction.
  • Qualitative Evaluation: The method successfully reconstructed thread curves in cases of no occlusion and finite self-occlusion (overhand knots).
  • Tool Occlusion: While detection performance deteriorates with tool occlusion, the method still achieved "human level" performance for threads with markers in these scenarios.
  • Speed: The method can generate thread reconstructions at 15 frames per second on their platform.

Limitations and Future Work

  • Tool Occlusion: The primary limitation acknowledged is the performance degradation in cases of significant tool occlusion, which can lead to incomplete gradient maps.
  • Sequential Information: The current framework does not leverage temporal information from video sequences.
  • Future Work: The authors plan to integrate time-specific constraints into their framework using recurrent neural networks (RNNs) to further improve robustness, especially in dynamic surgical scenarios. They also suggest that for certain tasks like knot recognition, understanding the overall scene context might be sufficient even if precise thread reconstruction is challenging.

Summary updated

Abstract

In robotic surgery, task automation and learning from demonstration combined with human supervision is an emerging trend for many new surgical robot platforms. One such task is automated anastomosis, which requires bimanual needle handling and suture detection. Due to the complexity of the surgical environment and varying patient anatomies, reliable suture detection is difficult, which is further complicated by occlusion and thread topologies. In this paper, we propose a multi-stage framework for suture thread detection based on deep learning. Fully convolutional neural networks are used to obtain the initial detection and the overlapping status of suture thread, which are later fused with the original image to learn a gradient road map of the thread. Based on the gradient road map, multiple segments of the thread are extracted and linked to form the whole thread using a curvilinear structure detector. Experiments on two different types of sutures demonstrate the accuracy of the proposed framework.
BibTeX
@inproceedings{hu2018multi,
  author = {Yang Hu and Yun Gu and Jie Yang and Guang-Zhong Yang},
  title = {{Multi-Stage Suture Detection for Robot Assisted Anastomosis Based on Deep Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2018},
  url = {https://openreview.net/forum?id=o0QB75a4UB},
}
Figure