Redefining the Laparoscopic Spatial Sense: AI-Based Intra- and Postoperative Measurement from Stereoimages

Leopold Müller, Patrick Hemmer, Moritz Queisner, Igor M. Sauer, Simeon Allmendinger, Johannes Jakubik, Michael Vössing, Niklas KühlAAAI 2024 · DOI

Abstract

Laparoscopic surgery offers significant patient benefits but restricts the surgeon's depth perception and tactile feedback. Precise measurement of anatomical structures or tumors currently requires cumbersome physical tools. This paper introduces a novel AI-driven pipeline that provides a "digital ruler" for surgeons. By leveraging deep learning for depth estimation (RAFT-Stereo) and tool tracking (YOLOv8), the system achieves sub-millimeter accuracy in 3D measurements using standard stereo-endoscope images. The method is robust against common surgical artifacts like smoke and reflections, supporting both intraoperative real-time tracking and postoperative analysis.

Introduction

In minimally invasive surgery, judging the size of a tumor or the width of a resection margin is notoriously difficult due to the loss of 3D spatial cues. Traditional measurement techniques often involve inserting physical rulers through trocars, which is inefficient and potentially traumatic. While computer vision solutions have been proposed, the surgical environment—characterized by textureless tissues, specular reflections, and obscurations from blood or smoke—presents a significant challenge for standard algorithms. This work aims to provide a passive, versatile measurement tool that integrates seamlessly into the surgical workflow without additional hardware.

Methodology

The proposed system transforms 2D stereo images into precise 3D measurements through a multi-stage pipeline.

The AI-based measurement pipeline, illustrating the flow from stereo input and camera parameters through AI processing to final distance outputs.
  1. Point Selection: The system operates in two modes. In offline mode, users manually select points on the screen. In online mode, a YOLOv8 model automatically detects surgical grasper tips, allowing the instruments themselves to serve as measurement probes.
  2. Disparity and Depth Estimation: To overcome the limitations of traditional Stereo Block Matching (SGBM) on textureless surfaces, the pipeline utilizes RAFT-Stereo. This AI architecture excels at finding correspondences in challenging visual conditions.
  3. Surface Reconstruction: The resulting 3D point cloud is processed using Poisson Surface Reconstruction (PSR) to create a watertight mesh.
  4. Distance Calculation: The system computes two primary metrics:
  5. Euclidean Distance: The direct straight-line distance between two points.
  6. On-surface Distance: The path along the tissue surface, calculated via Dijkstra’s algorithm and refined with spline interpolation to account for the mesh's discrete nature.

Experimental Results and Insights

The system was validated using a variety of datasets, including a custom 3D-printed "Playground" model with known ground truth, ex-vivo robotic surgery data, and real-world footage from the Hamlyn Centre Dataset.

Reconstructed point cloud demonstrating the calculation of different distance metrics across a complex surface.

Key findings include:

  • High Precision: The system achieved a Mean Absolute Error (MAE) of less than 0.3 mm for direct distances in offline mode. In online mode, using automatic tool tracking, the error remained under 1.5 mm.
  • Spline Refinement: Basic mesh-based paths often overestimate distance due to vertex "jaggedness." Spline interpolation significantly improved accuracy, reducing on-surface errors to approximately 0.24 mm in controlled tests.
  • AI Robustness: Unlike traditional CV methods that failed in low-texture or poorly lit environments, the RAFT-Stereo model maintained high accuracy even when processing monochromatic phantom models.

Discussion and Future Work

The integration of YOLOv8 effectively turns existing surgical tools into digital calipers, fulfilling a critical clinical need for non-invasive measurement. However, computational speed remains a hurdle; specifically, the PSR step currently lacks GPU acceleration, limiting real-time mesh updates.

Future developments will focus on "persistent reference points," allowing the system to track locations even when tools move out of frame. Additionally, training models to recognize specific anatomical structures could automate measurements like vessel diameters, further enhancing the surgeon's spatial awareness.

Summary updated

BibTeX
@inproceedings{muller2024redefining,
  author = {Leopold Müller and Patrick Hemmer and Moritz Queisner and Igor M. Sauer and Simeon Allmendinger and Johannes Jakubik and Michael Vössing and Niklas Kühl},
  title = {{Redefining the Laparoscopic Spatial Sense: AI-Based Intra- and Postoperative Measurement from Stereoimages}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30334},
}
Figure