1P1-T05 Surgical Thread Segmentation Using Kalman Filter for Surgical Task Automation(Robot Vision)

Takayuki Osa, Isao Oka, Naohiko Sugita, Mamoru MitsuishiThe Proceedings of JSME Annual Conference on Robotics and Mechatronics (Robomec) 2014 · DOI

Abstract

Surgical task automation is expected to improve the performance of robotic surgery. Although segmentation of surgical threads in laparoscopic images is one of prerequisites for surgical task automation, the surgical thread segmentation in an unstructured environment is a very challenging problem. This study aims to develop a scheme for surgical thread segmentation for surgical task automation. We propose a segmentation algorithm wherein the continuity of a thread is utilized using the Kalman filter. Once a user clicks one point of the surgical thread in the image, the system traces the thread from the clicked point and segments the rest of the thread automatically. The algorithm was tested using the images of the thread in both structured and unstructured environments.

Introduction

The advancement of robotic surgery toward full autonomy requires robust perception systems capable of identifying and tracking flexible surgical materials. Autonomous suturing, a key goal in surgical automation, necessitates the precise segmentation of surgical threads within laparoscopic video feeds. However, the laparoscopic environment is inherently "unstructured," characterized by clutter from surgical instruments, varying organ textures, and the presence of blood. These factors make traditional computer vision techniques struggle to distinguish thin, deformable threads from complex backgrounds. This paper presents a semi-automated approach that leverages the physical continuity of threads to perform segmentation starting from a single user-defined point.

Methodology

The proposed algorithm adopts a "trace-from-point" strategy, treating the thread as a continuous path rather than a set of independent pixels. This localized approach significantly reduces computational overhead and noise interference.

State Estimation via Kalman Filter

The core of the tracking mechanism is a Kalman Filter. Unlike simple coordinate trackers, this system maintains a state vector that includes the thread's position ($x, y$), its local direction ($\theta$), and its brightness intensity ($I$). By incorporating direction and brightness, the filter can predict the next segment of the thread with higher fidelity, assuming that these properties change smoothly along the thread's length.

Illustration of the search window and Kalman filter prediction process.

Template Matching and Search Optimization

To refine the Kalman Filter's prediction, the system performs a localized search. It utilizes four 5x5 pixel binary templates representing different thread orientations: horizontal, vertical, and two diagonal directions.

Binary templates used for orientation-aware matching.

The algorithm optimizes the search by determining the primary axis of movement. If the thread is moving more horizontally, the search is conducted along the x-axis; otherwise, it focuses on the y-axis. The best-matching template at the predicted location serves as the "measurement" to update the Kalman Filter's state.

Continuity and Stopping Criteria

To ensure the tracker does not deviate onto organ edges or blood vessels, a continuity constraint is applied. The system calculates the dot product between the predicted direction and the observed direction. If the angular change is too sharp or the brightness deviates significantly from the tracked state, the algorithm terminates the trace, identifying the end of the thread or an occlusion.

Experimental Evaluation

The algorithm was evaluated using two distinct datasets:

  1. Structured Environment: Threads placed on a uniform, high-contrast surgical drape.
  2. Unstructured Environment: Threads superimposed on actual laparoscopic images containing organs and surgical tools.

Results and Discussion

In structured environments, the algorithm demonstrated high reliability, successfully tracing threads from end to end. A key observation was the system's sensitivity to the initial starting point; a stable "warm-up" period for the Kalman Filter is necessary for consistent tracking.

In unstructured environments, performance was more variable. While the algorithm could navigate complex backgrounds, it occasionally failed at the boundaries of high-contrast features, such as the edges of blood vessels or organs. The reliance on grayscale information proved to be a significant limitation, as the intensity of the thread often matched the intensity of background structures.

Conclusion

This study demonstrates the feasibility of using a Kalman Filter-based tracing approach for surgical thread segmentation. By focusing on local continuity and directional consistency, the system can ignore much of the noise present in laparoscopic images. Future work will focus on transitioning from grayscale to the RGB color space to improve discriminative power and exploring more robust filtering techniques, such as Particle Filters, to better handle non-Gaussian noise in complex surgical scenes.

Summary updated

Abstract

Surgical task automation is expected to improve the performance of robotic surgery. Although segmentation of surgical threads in laparoscopic images is one of prerequisites for surgical task automation, the surgical thread segmentation in an unstructured environment is a very challenging problem. This study aims to develop a scheme for surgical thread segmentation for surgical task automation. We propose a segmentation algorithm wherein the continuity of a thread is utilized using the Kalman filter. Once a user clicks one point of the surgical thread in the image, the system traces the thread from the clicked point and segments the rest of the thread automatically. The algorithm was tested using the images of the thread in both structured and unstructured environments.
BibTeX
@inproceedings{osa2014p,
  author = {Takayuki Osa and Isao Oka and Naohiko Sugita and Mamoru Mitsuishi},
  title = {{1P1-T05 Surgical Thread Segmentation Using Kalman Filter for Surgical Task Automation(Robot Vision)}},
  booktitle = {The Proceedings of JSME Annual Conference on Robotics and Mechatronics (Robomec)},
  year = {2014},
  doi = {10.1299/jsmermd.2014.\_1p1-t05\_1},
}
Figure