Real-time image-based instrument classification for laparoscopic surgery

Sebastian Bodenstedt, Antonia Ohnemus, Darko Katic, Anna-Laura Wekerle, Martin Wagner, Hannes Kenngott, Beat P. Müller-Stich, Rüdiger Dillmann, Stefanie SpeidelarXiv 2018 · arXiv

Abstract

This paper introduces a real-time, image-only system for detecting and identifying surgical instruments during laparoscopic procedures. By combining GPU-accelerated pixel-wise segmentation with a cascade of Random Forest classifiers, the system achieves instrument detection rates of up to 86%. This framework provides a foundation for context-aware surgical assistance without the need for external sensors or markers.

Introduction

In laparoscopic surgery, surgeons operate through small incisions using long instruments and a camera feed. To provide "context-aware" assistance—such as displaying relevant anatomical data or predicting the next step of an operation—a computer system must first understand the scene. A critical component of this understanding is identifying which tools are currently in use. While previous research focused on locating tools, classifying their specific type (e.g., grasper vs. clip applier) in real-time is significantly harder. This paper addresses the challenge of identifying tool types using only raw video frames, avoiding the impracticality of specialized markers or sensors.

Methodology

The authors propose a multi-stage pipeline designed for speed and robustness, as illustrated in the system architecture.

System architecture showing the multi-stage pipeline from mask generation to instrument identification.

1. Mask Generation

Laparoscopic images are circular. The system automatically detects and masks the black borders to prevent them from interfering with color-based features.

2. Instrument Detection

  • Segmentation: A Random Forest classifier performs pixel-wise segmentation on the GPU, analyzing color spaces (RGB, HSV, LAB) and gradients to find "instrument-like" pixels.
  • Refinement: The system uses morphological operations to close gaps (such as blood on a tool shaft) and groups contours based on their principal direction (eigen-decomposition) to form bounding boxes.

3. Instrument Identification (The Cascade)

To maintain real-time speeds, the system uses a two-stage "cascade" of Random Forests:

  • Stage 1 (Binary): A fast classifier uses simple hue and saturation histograms to decide if a bounding box actually contains an instrument.
  • Stage 2 (Multiclass): If the box passes Stage 1, a more complex classifier uses rotation-invariant gradient histograms and PCA-reduced SURF (Speeded-Up Robust Features) to determine the specific tool type.
Visual output of the detection pipeline: (a) original frames, (b) segmentation masks, and (c) bounding box extraction.

Experimental Setup

The study utilized five laparoscopic videos from adrenalectomies and pancreas resections. The system was trained to recognize four common instruments (LigaSure, atraumatic grasper, aspirator, clip applier) plus "background" and "unknown" classes. Evaluation was performed on two sets: 411 manually selected frames with clear visibility and 650 frames representing a realistic, "noisy" surgical workflow.

Key Results

  • Detection Performance: The system successfully located up to 86% of instruments in realistic video sequences.
  • Classification Accuracy: On automatically detected boxes, the recognition rate was 49% (rising to 58% on manually provided boxes).
  • Speed: The detection framework runs at approximately 22.2Hz, and the identification stage adds about 32.5Hz. Combined, the system operates at a usable real-time frequency.
  • Rotation Invariance: The authors achieved rotation invariance by calculating gradient histograms and shifting the bins so the highest peak always sits at the first position.

Discussion and Limitations

A primary challenge identified is the "Shaft Problem": most surgical instruments have very similar metallic shafts, with distinguishing features located only at the tips. If the tip is buried in tissue or covered in blood, classification accuracy drops significantly.

Furthermore, because the system analyzes each frame in isolation, it struggles with occlusions, motion blur, and overlapping instruments. Future work will incorporate temporal tracking to "remember" a tool's identity from previous frames where the tip was visible, maintaining accurate classification even when the tool is partially obscured.

Summary updated

Abstract

During laparoscopic surgery, context-aware assistance systems aim to alleviate some of the difficulties the surgeon faces. To ensure that the right information is provided at the right time, the current phase of the intervention has to be known. Real-time locating and classification the surgical tools currently in use are key components of both an activity-based phase recognition and assistance generation. In this paper, we present an image-based approach that detects and classifies tools during laparoscopic interventions in real-time. First, potential instrument bounding boxes are detected using a pixel-wise random forest segmentation. Each of these bounding boxes is then classified using a cascade of random forest. For this, multiple features, such as histograms over hue and saturation, gradients and SURF feature, are extracted from each detected bounding box. We evaluated our approach on five different videos from two different types of procedures. We distinguished between the four most common classes of instruments (LigaSure, atraumatic grasper, aspirator, clip applier) and background. Our method succesfully located up to 86% of all instruments respectively. On manually provided bounding boxes, we achieve a instrument type recognition rate of up to 58% and on automatically detected bounding boxes up to 49%. To our knowledge, this is the first approach that allows an image-based classification of surgical tools in a laparoscopic setting in real-time.
BibTeX
@misc{bodenstedt2018real,
  author = {Sebastian Bodenstedt and Antonia Ohnemus and Darko Katic and Anna-Laura Wekerle and Martin Wagner and Hannes Kenngott and Beat P. Müller-Stich and Rüdiger Dillmann and Stefanie Speidel},
  title = {{Real-time image-based instrument classification for laparoscopic surgery}},
  year = {2018},
  eprint = {1808.00178},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.00178},
}
Figure