AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy

Ziyi Wang, Bo Lu, Yonghao Long, Fangxun Zhong, Tak Hong Cheung, Qi Dou, Yunhui Liu

Introduction

In the field of Computer-Assisted Intervention (CAI), significant progress has been made in individual image analysis tasks. However, achieving true surgical automation—where robotic systems autonomously manage camera field-of-view (FoV) or assist in tissue manipulation—remains a significant bottleneck. This challenge is largely attributed to a lack of comprehensive, multi-task datasets that capture the complex, context-dependent nature of real-world surgery.

Existing surgical datasets often suffer from three primary limitations: a focus on single tasks (e.g., only workflow or only segmentation), limited scale, and a lack of clinical relevance to complex procedures like laparoscopic hysterectomy. To address these gaps, the AutoLaparo dataset provides a large-scale, multi-task benchmark designed to foster the development of "context-aware" robotic assistants capable of navigating the intricacies of gynecological surgery.

The AutoLaparo Dataset

AutoLaparo is the first dataset of its kind to integrate synchronized annotations for surgical workflow, laparoscope motion, and instrument/anatomy segmentation within the context of laparoscopic hysterectomy. The dataset is derived from 21 full-length surgical videos, totaling 1,388 minutes of footage recorded at 25 fps.

To ensure high clinical fidelity, all annotations were performed by a senior gynecologist with over 30 years of experience and a specialist. The dataset is publicly available to the research community at https://autolaparo.github.io.

Integrated Multi-Task Framework

The researchers formulated a three-tier annotation strategy to provide a holistic understanding of the surgical environment:

1. Surgical Workflow Recognition

The procedure is divided into seven distinct clinical phases, such as Dividing Uterine Vessels and Suturing. Recognizing these phases is critical for an AI assistant to understand the broader context of the operation.

2. Laparoscope Motion Prediction

This task involves identifying seven types of camera movement: Up, Down, Left, Right, Zoom-in, Zoom-out, and Static. The dataset includes 300 selected 10-second clips specifically for this purpose. By analyzing the 5 seconds of footage preceding a camera movement, models are trained to recognize "non-optimal views" and predict the necessary adjustments to reach an "optimal view."

3. Instrument and Key Anatomy Segmentation

The dataset provides 1,800 frames containing 5,936 pixel-wise masks across 9 classes. This includes the uterus and four types of surgical instruments. A key feature of this task is the distinction between the shaft and the manipulator (the functional head) of the tools, which is essential for accurate tracking and 3D pose estimation.

Key Insights and Benchmarks

The authors established baseline benchmarks using state-of-the-art (SOTA) models to evaluate the difficulty of each task:

  • Workflow Recognition: Utilizing Trans-SVNet, the models achieved approximately 82% accuracy. This suggests that temporal transformers are highly effective at capturing the sequential nature of surgical phases.
  • Segmentation: Using YOLACT, the researchers achieved a Mask Average Precision (AP) of 54.09%. While promising, the complex environment of the abdominal cavity—characterized by blood, smoke, and varying lighting—remains a significant challenge for standard computer vision models.
  • Laparoscope Motion Prediction: This proved to be the most difficult task, with a ResNet-50 + LSTM architecture achieving only ~29.8% accuracy. This low performance indicates that visual data alone is insufficient; predicting camera movement likely requires a deeper understanding of "surgical intent" and the integration of workflow context.

Discussion and Future Work

The AutoLaparo dataset highlights the interconnectedness of surgical tasks. For instance, knowing the current surgical phase (Task 1) narrows the likely instruments and anatomy visible (Task 3), which in turn informs where the camera should be directed (Task 2).

While AutoLaparo represents a significant step forward, the authors acknowledge that 21 videos is still a relatively small sample size compared to general computer vision datasets. Future work will focus on expanding the dataset with more cases and introducing additional tasks, such as 3D pose estimation and instrument tracking, to further the goal of full surgical automation.

Summary updated

BibTeX
@inproceedings{wang2022autolaparo,
  author = {Ziyi Wang and Bo Lu and Yonghao Long and Fangxun Zhong and Tak Hong Cheung and Qi Dou and Yunhui Liu},
  title = {{AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2022},
  doi = {10.1007/978-3-031-16449-1\_46},
}
Figure