Abstract
Laparoscopic surgery offers significant patient benefits but restricts the surgeon's depth perception and tactile feedback. Precise measurement of anatomical structures or tumors currently requires cumbersome physical tools. This paper introduces a novel AI-driven pipeline that provides a "digital ruler" for surgeons. By leveraging deep learning for depth estimation (RAFT-Stereo) and tool tracking (YOLOv8), the system achieves sub-millimeter accuracy in 3D measurements using standard stereo-endoscope images. The method is robust against common surgical artifacts like smoke and reflections, supporting both intraoperative real-time tracking and postoperative analysis.
Introduction
In minimally invasive surgery, judging the size of a tumor or the width of a resection margin is notoriously difficult due to the loss of 3D spatial cues. Traditional measurement techniques often involve inserting physical rulers through trocars, which is inefficient and potentially traumatic. While computer vision solutions have been proposed, the surgical environment—characterized by textureless tissues, specular reflections, and obscurations from blood or smoke—presents a significant challenge for standard algorithms. This work aims to provide a passive, versatile measurement tool that integrates seamlessly into the surgical workflow without additional hardware.
Methodology
The proposed system transforms 2D stereo images into precise 3D measurements through a multi-stage pipeline.
- Point Selection: The system operates in two modes. In offline mode, users manually select points on the screen. In online mode, a YOLOv8 model automatically detects surgical grasper tips, allowing the instruments themselves to serve as measurement probes.
- Disparity and Depth Estimation: To overcome the limitations of traditional Stereo Block Matching (SGBM) on textureless surfaces, the pipeline utilizes RAFT-Stereo. This AI architecture excels at finding correspondences in challenging visual conditions.
- Surface Reconstruction: The resulting 3D point cloud is processed using Poisson Surface Reconstruction (PSR) to create a watertight mesh.
- Distance Calculation: The system computes two primary metrics:
- Euclidean Distance: The direct straight-line distance between two points.
- On-surface Distance: The path along the tissue surface, calculated via Dijkstra’s algorithm and refined with spline interpolation to account for the mesh's discrete nature.
Experimental Results and Insights
The system was validated using a variety of datasets, including a custom 3D-printed "Playground" model with known ground truth, ex-vivo robotic surgery data, and real-world footage from the Hamlyn Centre Dataset.
Key findings include:
- High Precision: The system achieved a Mean Absolute Error (MAE) of less than 0.3 mm for direct distances in offline mode. In online mode, using automatic tool tracking, the error remained under 1.5 mm.
- Spline Refinement: Basic mesh-based paths often overestimate distance due to vertex "jaggedness." Spline interpolation significantly improved accuracy, reducing on-surface errors to approximately 0.24 mm in controlled tests.
- AI Robustness: Unlike traditional CV methods that failed in low-texture or poorly lit environments, the RAFT-Stereo model maintained high accuracy even when processing monochromatic phantom models.
Discussion and Future Work
The integration of YOLOv8 effectively turns existing surgical tools into digital calipers, fulfilling a critical clinical need for non-invasive measurement. However, computational speed remains a hurdle; specifically, the PSR step currently lacks GPU acceleration, limiting real-time mesh updates.
Future developments will focus on "persistent reference points," allowing the system to track locations even when tools move out of frame. Additionally, training models to recognize specific anatomical structures could automate measurements like vessel diameters, further enhancing the surgeon's spatial awareness.
Summary updated