Surgical Skill Assessment on In-Vivo Clinical Data via the Clearness of Operating Field

Daochang Liu, Tingting Jiang, Yizhou Wang, Rulin Miao, Fei Shan, Ziyu Li

Abstract

Surgical skill assessment is important for surgery training and quality control. Prior works on this task largely focus on basic surgical tasks such as suturing and knot tying performed in simulation settings. In contrast, surgical skill assessment is studied in this paper on a real clinical dataset, which consists of fifty-seven in-vivo laparoscopic surgeries and corresponding skill scores annotated by six surgeons. From analyses on this dataset, the clearness of operating field (COF) is identified as a good proxy for overall surgical skills, given its strong correlation with overall skills and high inter-annotator consistency. Then an objective and automated framework based on neural network is proposed to predict surgical skills through the proxy of COF. The neural network is jointly trained with a supervised regression loss and an unsupervised rank loss. In experiments, the proposed method achieves 0.55 Spearman's correlation with the ground truth of overall technical skill, which is even comparable with the human performance of junior surgeons.

Introduction

Traditional surgical skill assessment is a subjective and labor-intensive process, typically requiring expert surgeons to review video recordings. While automated systems have emerged, they often rely on "benchtop" simulations—such as suturing synthetic materials—and specialized motion-tracking hardware. These methods face a significant "simulation gap," as real-world clinical surgeries involve unpredictable patient conditions, bleeding, and high-stress environments that simulations cannot replicate.

Furthermore, motion-based metrics (tracking tools or hands) are difficult to implement in real operating rooms. Robotic kinematics are often proprietary, external sensors can disrupt workflows, and visual tracking frequently fails due to occlusions or poor lighting. This research explores whether skill can be assessed using only the visual information present in standard laparoscopic video.

Defining the Proxy: Clearness of Operating Field (COF)

The researchers identify the Clearness of Operating Field (COF) as a robust proxy for overall surgical skill. COF measures a surgeon's ability to manage bleeding and maintain clear visibility of anatomical landmarks. Unlike more abstract metrics like "gentleness," COF correlates strongly with overall technical skill and demonstrates high inter-annotator consistency among experts.

Visual comparison between good and poor clearness of operating field (COF) in clinical laparoscopic surgery.

Automated Assessment Framework

The proposed framework uses a video-based neural network to predict skill scores. The pipeline involves:

  1. Feature Extraction: The system extracts color features (histograms in RGB, HSV, and Red-ratio spaces) to detect bleeding and semantic features using a ResNet-101 model for high-level visual context.
  2. Normalization: To account for variations in lighting and camera equipment, color features are normalized against the initial 30% of the video, where the field is typically cleanest.
  3. Dual-Branch Model: A "Score Branch" evaluates individual frames, while a "Weight Branch" learns to prioritize frames that are most indicative of skill (e.g., active dissection vs. camera cleaning).
  4. Hybrid Loss Function: The model is trained using a supervised regression loss against expert COF scores and an unsupervised rank loss. The rank loss leverages the temporal logic that COF generally degrades as a procedure progresses and blood accumulates.

Experimental Results

The study utilized a new dataset of 57 in-vivo laparoscopic gastrectomy surgeries, annotated by three senior and three junior surgeons. The automated method achieved a 0.55 Spearman’s correlation with the ground truth for Overall Technical Skill (OTS). This performance is comparable to that of junior surgeons, whose correlations ranged from 0.41 to 0.62.

The model provides temporal feedback, identifying specific 'error moves' and critical steps through frame-level scores and weights.

Beyond a single score, the model provides granular feedback. As shown in the temporal analysis, the system can pinpoint "error moves" where the COF score drops sharply or highlight critical steps where the frame weight increases, indicating high informational value for skill assessment.

Conclusion

By shifting the focus from motion tracking to the visual result of surgical actions—specifically the clearness of the operating field—this framework provides a scalable, hardware-free solution for clinical skill assessment. Future work will focus on validating the model across diverse surgical procedures and incorporating more advanced temporal modeling to capture the full narrative of a surgery.

Summary updated

Abstract

Surgical skill assessment is important for surgery training and quality control. Prior works on this task largely focus on basic surgical tasks such as suturing and knot tying performed in simulation settings. In contrast, surgical skill assessment is studied in this paper on a real clinical dataset, which consists of fifty-seven in-vivo laparoscopic surgeries and corresponding skill scores annotated by six surgeons. From analyses on this dataset, the clearness of operating field (COF) is identified as a good proxy for overall surgical skills, given its strong correlation with overall skills and high inter-annotator consistency. Then an objective and automated framework based on neural network is proposed to predict surgical skills through the proxy of COF. The neural network is jointly trained with a supervised regression loss and an unsupervised rank loss. In experiments, the proposed method achieves 0.55 Spearman's correlation with the ground truth of overall technical skill, which is even comparable with the human performance of junior surgeons.
BibTeX
@inproceedings{liu2019surgical,
  author = {Daochang Liu and Tingting Jiang and Yizhou Wang and Rulin Miao and Fei Shan and Ziyu Li},
  title = {{Surgical Skill Assessment on In-Vivo Clinical Data via the Clearness of Operating Field}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  url = {https://openreview.net/forum?id=CqIL7QB5mFk},
}
Figure