Surgical Skill Assessment on In-Vivo Clinical Data via the Clearness of Operating Field

Daochang Liu, Tingting Jiang, Yizhou Wang, Rulin Miao, Fei Shan, Ziyu Li

Evaluating a surgeon’s skill is traditionally a subjective, labor-intensive process. While automated systems have attempted to solve this, they usually rely on "benchtop" simulations—think suturing a piece of rubber in a lab—and complex motion-tracking hardware. But how do these systems perform in a real, bloody, high-stakes operating room?

In their paper, "Surgical Skill Assessment on In-Vivo Clinical Data via the Clearness of Operating Field," researchers from Peking University and associated hospitals move beyond the lab. They propose a new way to measure skill in real-world clinical data by looking at one specific proxy: how clean the surgeon keeps the "operating field."

TL;DR

The researchers developed an automated framework to assess surgical skill in real-world laparoscopic videos by using the Clearness of Operating Field (COF) as a proxy. By training a neural network on color and semantic features from 57 clinical surgeries, they achieved skill assessment performance comparable to junior surgeons without requiring specialized motion-tracking hardware.

What Problem Does This Paper Address?

Most automated surgical assessment research relies on datasets like JIGSAWS, which feature basic tasks (suturing, knot-tying) performed on simulators. However, there is a massive "simulation gap." Real clinical surgeries involve varying patient conditions, bleeding, and high stress levels that simulations can't replicate.

Furthermore, existing methods usually rely on motion-based metrics (tracking tools or hands). In a real operating room, getting clean motion data is a nightmare: robotic kinematics are only available for expensive robots, external sensors interrupt the workflow, and visual tracking often fails due to occlusions or poor lighting. This paper asks: Can we assess skill using only the visual information already present in standard surgery videos?

Methods

The authors propose a video-based approach that predicts surgical skill by focusing on the Clearness of Operating Field (COF)—essentially, how well the surgeon manages bleeding and maintains visibility of anatomical landmarks.

The Pipeline:

  1. Feature Extraction: The system extracts two types of data from video frames:
  2. Color Features: Histograms in RGB, HSV, and "Red-ratio" spaces to detect the presence and severity of bleeding.
  3. Semantic Features: High-level visual information extracted using a ResNet-101 model.
  4. Normalization: To handle different cameras and lighting, they normalize color features against the first 30% of each video (assuming the field is cleanest at the start).
  5. The Model: A dual-branch neural network:
  6. Score Branch: Evaluates the quality of individual frames.
  7. Weight Branch: Learns which frames are most important for the final score (e.g., a frame during an active dissection is more telling than a frame where the camera is being cleaned).
  8. The Loss Functions:
    • Supervised Regression: Training against expert-provided COF scores.
  9. Unsupervised Rank Loss: A clever addition based on the observation that COF generally degrades over time as blood accumulates. The model is penalized if it doesn't rank the "start" of the surgery as clearer than the "end."

Datasets

The study introduces a new clinical dataset consisting of 57 in-vivo laparoscopic gastrectomy surgeries (for gastric cancer).

  • Annotation: Six surgeons (3 seniors, 3 juniors) annotated the videos across 14 metrics, including standard OSATS (Objective Structured Assessment of Technical Skill) and new procedure-specific metrics.
  • Significance: Unlike previous clinical datasets that only featured short clips or very few videos, this dataset contains full-length, complex procedures.

Key Insights and Ideas

  • COF as a "Golden Proxy": The most significant insight is that COF correlates more strongly with overall technical skill than almost any other metric. It also showed the highest consistency between different expert annotators, making it a more "objective" target for AI than "gentleness" or "flow of operation."
  • Appearance over Motion: By focusing on the result of the surgeon's actions (a clear field) rather than the path of their hands (motion), the system becomes much more robust to the visual chaos of a real surgery.
  • Temporal Logic: The use of "Rank Loss" acknowledges the reality of surgery—that bleeding is cumulative. This allows the model to learn from the temporal structure of the procedure without needing a human to label every single frame.

Key Results

  • Skill Correlation: The proposed method achieved a 0.55 Spearman’s correlation with the ground truth for Overall Technical Skill (OTS).
  • Human-Level Performance: This 0.55 correlation is comparable to the performance of junior surgeons (who achieved 0.41 to 0.62 in various tests).
  • Proxy Power: Predicting overall skill via the COF proxy performed significantly better than trying to predict overall skill scores directly from the video.
  • Feedback: The model can generate frame-level scores, allowing it to point out specific "error moves" (where the score drops) or critical steps (where the frame weight increases).

Limitations and Future Work

  • Single Surgeon/Procedure: The current dataset features only one primary surgeon and one type of procedure (gastrectomy). The authors acknowledge that the model needs to be validated across different surgeons and surgery types.
  • Temporal Modeling: While the rank loss helps, the model currently processes frames somewhat independently. Future work could incorporate more sophisticated temporal modeling (like LSTMs or Transformers) to understand the "story" of the surgery.
  • Domain Knowledge: The authors suggest that explicitly modeling surgical steps and human anatomy could further bridge the gap between AI and expert human assessment.

Summary updated

Abstract

Surgical skill assessment is important for surgery training and quality control. Prior works on this task largely focus on basic surgical tasks such as suturing and knot tying performed in simulation settings. In contrast, surgical skill assessment is studied in this paper on a real clinical dataset, which consists of fifty-seven in-vivo laparoscopic surgeries and corresponding skill scores annotated by six surgeons. From analyses on this dataset, the clearness of operating field (COF) is identified as a good proxy for overall surgical skills, given its strong correlation with overall skills and high inter-annotator consistency. Then an objective and automated framework based on neural network is proposed to predict surgical skills through the proxy of COF. The neural network is jointly trained with a supervised regression loss and an unsupervised rank loss. In experiments, the proposed method achieves 0.55 Spearman’s correlation with the ground truth of overall technical skill, which is even comparable with the human performance of junior surgeons.
BibTeX
@inproceedings{liu2019surgical,
  author = {Daochang Liu and Tingting Jiang and Yizhou Wang and Rulin Miao and Fei Shan and Ziyu Li},
  title = {{Surgical Skill Assessment on In-Vivo Clinical Data via the Clearness of Operating Field}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  url = {https://openreview.net/forum?id=CqIL7QB5mFk},
}

Loading related content...

Figure