LungSurg: A Generative AI System for Segmentation and Phase Classification in Thoracoscopic Lobectomy

Hengrui Liang, Zeping Yan, Yudong Zhang, Keyao Dai, Hongyan Li, Jiali Shen, P. P. Li, Jipeng Jiang, Guochao Zhang, X. Zhang, Hao Chen, Honglang Zhang, Yuzhuo Zhang, Shujun Liang, Minsheng Chen, Xin Wang, Anyi Rao, W. Wang, Lei Zhao, Yuchen Guo, Jianxing HeMedComm 2026 · DOI

TL;DR

This paper introduces LungSurg, an AI system that can accurately identify anatomical structures and surgical instruments, and classify surgical phases during video-assisted thoracoscopic surgery (VATS) lobectomy. LungSurg not only performs comparably to experienced surgeons in anatomical identification but also shows promise in improving surgical resident training.

What Problem Does This Paper Address?

Video-assisted thoracoscopic surgery (VATS) lobectomy, a standard treatment for early-stage lung cancer, is technically demanding. Surgeons face a steep learning curve due to complex thoracic anatomy, inter-patient variability, and intraoperative challenges like occlusions from smoke or instruments. This makes it difficult for residents to master the procedure and for experienced surgeons to maintain consistent performance. Existing AI solutions have primarily focused on less complex surgeries, leaving a gap in applying advanced AI to high-stakes thoracic procedures like VATS lobectomy.

Methods

LungSurg is built upon two interconnected neural networks:

  • SurgSeg (Segmentation Network): This network identifies intrathoracic anatomy and surgical instruments. It utilizes a Mask R-CNN architecture with a ResNet-50 backbone, enhanced by non-local (NL) modules and multi-head self-attention (MHSA) to capture long-range dependencies and improve performance. A novel aspect is the use of generative pre-training with image inpainting, inspired by Masked Autoencoders (MAE). This self-supervised approach allows the model to learn robust feature representations from unlabeled surgical videos by reconstructing occluded image parts, effectively building a "visual knowledge base" before fine-tuning on annotated data. A co-occurrence loss function is also employed to leverage contextual dependencies between surgical elements.
  • SurgClass (Classification Network): This network recognizes surgical phases. It integrates information from SurgSeg, including per-pixel confidence scores, dot products between images and segmentation masks, and segmentation features. It uses a combination of 3D action recognition networks and bidirectional long short-term memory (bi-LSTM) networks to model both spatial-temporal patterns and temporal dependencies.

Datasets

The study utilized a multicenter dataset of 222 integrated VATS lobectomy videos collected from eight centers between January 1, 2018, and January 1, 2024. These videos encompassed complete surgical procedures.

  • Training Set: 165 videos from four centers.
  • Validation Set: 57 videos from other centers for external validation.

Over 32,000 annotations were created for anatomical structures and instruments, and over one million frames were annotated with phase information, representing 14 distinct surgical phases. The dataset was curated to exclude highly complex cases to establish a robust baseline.

Key Insights and Ideas

  • Generative Pre-training for Data Scarcity: The use of image inpainting for self-supervised pre-training is a significant contribution, enabling the development of high-performance AI models for specialized surgical tasks even with limited annotated data. This approach builds a foundational understanding of surgical scenes without requiring manual labels for every element.
  • Bridging the Gap to Complex Surgery: LungSurg demonstrates that advanced AI can be effectively applied to technically demanding procedures like VATS lobectomy, overcoming challenges like anatomical variability and intraoperative occlusions.
  • AI as a Training Tool: The pilot study on surgical residents highlights LungSurg's potential to revolutionize surgical education, leading to significant improvements in anatomical identification and phase classification skills compared to conventional methods.
  • Expert-Level Performance: LungSurg achieves performance comparable to senior surgeons in anatomical identification and even surpasses them in sensitivity, suggesting its utility as a safety adjunct.

Key Results

  • Segmentation Performance:
    • External validation: Mean Average Precision (mAP) of 0.745 for the left lung and 0.726 for the right lung.
  • Anatomical structures: mAP of 0.670 (left) and 0.679 (right).
  • Surgical instruments: mAP of 0.806 (left) and 0.755 (right).
  • Multi-center trained models significantly outperformed single-center trained models.
  • Classification Performance:
    • External validation: Top-1 accuracy of 71.5% and Top-3 accuracy of 88.0% for identifying 14 surgical phases.
  • Key procedural steps like bronchovascular management achieved high accuracies (Top-1: 0.678-0.796, Top-3: 0.804-0.988).
  • Human Expert Comparison:
    • LungSurg's detection rate (PID) was comparable to senior surgeons (0.634 vs. 0.636).
  • LungSurg achieved a higher sensitivity (SID) of 0.777, outperforming senior surgeons by 0.244.
  • Educational Study:
    • Experimental group (trained with LungSurg) showed significantly higher accuracy in anatomical identification (73.40% vs. 63.0%) and surgical instrument identification (88.20% vs. 79.60%) compared to the control group.
  • Surgical phase classification accuracy was also significantly higher for the experimental group (79.2% vs. 65.4%).

Limitations and Future Work

  • Generalizability and Robustness: The model's performance might degrade in scenarios with challenging visual conditions like lens blur, heavy smoke, or rapid instrument movements. Future work will focus on enhancing resilience through advanced image processing and temporal consistency checks. The initial dataset excluded highly complex cases, so future work will involve expanding the dataset and developing specialized modules for these scenarios.
  • Real-time Intraoperative Guidance: Transitioning LungSurg from an offline analysis tool to a real-time intraoperative guidance system is a major future direction. This requires significant algorithm optimization for low latency (e.g., pruning, quantization) and seamless integration with existing surgical platforms, along with the design of intuitive user interfaces.
  • Educational Validation: The educational study was a pilot, single-institution trial. Larger, multi-center trials are needed to confirm long-term skill retention and transfer to practical performance.

Summary updated

Abstract

The integration of artificial intelligence (AI) into surgical practices is advancing towards greater intelligence and precision. This study assesses the potential of AI in video-assisted thoracoscopic surgery (VATS) lobectomy for lung cancer by developing an AI system named LungSurg. LungSurg comprises two interconnected networks: a segmentation network for identifying intrathoracic anatomy and surgical instruments, and a classification network for recognizing surgical phases. We prospectively collected 222 VATS lobectomy videos from eight centers, generating over 32,000 annotations and more than one million frames with phase information. In external validation, the segmentation network achieved mean Average precision scores of 0.745 for the left lung and 0.726 for the right lung across various instruments and anatomical structures. The classification network demonstrated Top-1 and Top-3 accuracies of 71.5% and 88.0%, respectively, in identifying 14 surgical phases. Comparative experiments revealed that LungSurg performed comparably to senior surgeons in anatomical identification and surpassed them in sensitivity. In addition, an educational study showed that surgical residents trained with LungSurg significantly improved their anatomical identification and phase classification skills compared to those using conventional methods. These results indicate that LungSurg accurately analyzes VATS lobectomy procedures, highlighting the feasibility and potential of AI-driven tools in enhancing thoracic surgical practices.
BibTeX
@inproceedings{liang2026lungsurg,
  author = {Hengrui Liang and Zeping Yan and Yudong Zhang and Keyao Dai and Hongyan Li and Jiali Shen and P. P. Li and Jipeng Jiang and Guochao Zhang and X. Zhang and Hao Chen and Honglang Zhang and Yuzhuo Zhang and Shujun Liang and Minsheng Chen and Xin Wang and Anyi Rao and W. Wang and Lei Zhao and Yuchen Guo and Jianxing He},
  title = {{LungSurg: A Generative AI System for Segmentation and Phase Classification in Thoracoscopic Lobectomy}},
  booktitle = {MedComm},
  year = {2026},
  doi = {10.1002/mco2.70613},
}
Figure