Abstract
Surgical task automation is expected to improve the performance of robotic surgery. Although segmentation of surgical threads in laparoscopic images is one of prerequisites for surgical task automation, the surgical thread segmentation in an unstructured environment is a very challenging problem. This study aims to develop a scheme for surgical thread segmentation for surgical task automation. We propose a segmentation algorithm wherein the continuity of a thread is utilized using the Kalman filter. Once a user clicks one point of the surgical thread in the image, the system traces the thread from the clicked point and segments the rest of the thread automatically. The algorithm was tested using the images of the thread in both structured and unstructured environments.
Introduction
The advancement of robotic surgery toward full autonomy requires robust perception systems capable of identifying and tracking flexible surgical materials. Autonomous suturing, a key goal in surgical automation, necessitates the precise segmentation of surgical threads within laparoscopic video feeds. However, the laparoscopic environment is inherently "unstructured," characterized by clutter from surgical instruments, varying organ textures, and the presence of blood. These factors make traditional computer vision techniques struggle to distinguish thin, deformable threads from complex backgrounds. This paper presents a semi-automated approach that leverages the physical continuity of threads to perform segmentation starting from a single user-defined point.
Methodology
The proposed algorithm adopts a "trace-from-point" strategy, treating the thread as a continuous path rather than a set of independent pixels. This localized approach significantly reduces computational overhead and noise interference.
State Estimation via Kalman Filter
The core of the tracking mechanism is a Kalman Filter. Unlike simple coordinate trackers, this system maintains a state vector that includes the thread's position ($x, y$), its local direction ($\theta$), and its brightness intensity ($I$). By incorporating direction and brightness, the filter can predict the next segment of the thread with higher fidelity, assuming that these properties change smoothly along the thread's length.
Template Matching and Search Optimization
To refine the Kalman Filter's prediction, the system performs a localized search. It utilizes four 5x5 pixel binary templates representing different thread orientations: horizontal, vertical, and two diagonal directions.
The algorithm optimizes the search by determining the primary axis of movement. If the thread is moving more horizontally, the search is conducted along the x-axis; otherwise, it focuses on the y-axis. The best-matching template at the predicted location serves as the "measurement" to update the Kalman Filter's state.
Continuity and Stopping Criteria
To ensure the tracker does not deviate onto organ edges or blood vessels, a continuity constraint is applied. The system calculates the dot product between the predicted direction and the observed direction. If the angular change is too sharp or the brightness deviates significantly from the tracked state, the algorithm terminates the trace, identifying the end of the thread or an occlusion.
Experimental Evaluation
The algorithm was evaluated using two distinct datasets:
- Structured Environment: Threads placed on a uniform, high-contrast surgical drape.
- Unstructured Environment: Threads superimposed on actual laparoscopic images containing organs and surgical tools.
Results and Discussion
In structured environments, the algorithm demonstrated high reliability, successfully tracing threads from end to end. A key observation was the system's sensitivity to the initial starting point; a stable "warm-up" period for the Kalman Filter is necessary for consistent tracking.
In unstructured environments, performance was more variable. While the algorithm could navigate complex backgrounds, it occasionally failed at the boundaries of high-contrast features, such as the edges of blood vessels or organs. The reliance on grayscale information proved to be a significant limitation, as the intensity of the thread often matched the intensity of background structures.
Conclusion
This study demonstrates the feasibility of using a Kalman Filter-based tracing approach for surgical thread segmentation. By focusing on local continuity and directional consistency, the system can ignore much of the noise present in laparoscopic images. Future work will focus on transitioning from grayscale to the RGB color space to improve discriminative power and exploring more robust filtering techniques, such as Particle Filters, to better handle non-Gaussian noise in complex surgical scenes.
Summary updated