Surgical Procedural Planning as 3D World Modelling: Towards Automated Pulmonary Resection

Zhen Zhang, Zhaorong Dong, Xiao Yang, Liqin Huang, Qiang Wu, Taidui Zeng, Hanyu Zheng, Mingjing Yang, Shaohua Zheng, Wangbin Ding, Lin PanCVPR 2026 Findings 2026 · OpenReview

TL;DR

This paper introduces the first end-to-end 3D medical world model for automated surgical procedural planning in pulmonary resection. By combining a 3D vision-language model for perception, a conditional diffusion model for simulating surgical consequences, and a clinically grounded decision engine, the framework generates safe, patient-specific surgical sequences directly from raw CT scans.

What Problem Does This Paper Address?

Performing segmentectomy or subsegmentectomy (removing small, precise portions of the lung) is a highly effective way to treat early-stage lung cancer while preserving lung function. However, planning these surgeries is incredibly difficult. The branching pathways of arteries, veins, and bronchi vary wildly from patient to patient, and tumors are often tucked away near ambiguous landmarks.

Currently, surgeons rely on static 3D visualizations and must mentally simulate how each surgical action—like cutting a specific vessel—will alter the anatomy and impact the remaining lung tissue. Existing AI approaches fail to help because they treat anatomy as static geometry, lack the ability to forecast 3D anatomical outcomes, and struggle to integrate complex clinical reasoning into an end-to-end decision framework.

Methods

To bridge this gap, the authors propose a closed-loop "Perceive–Simulate–Decide" architecture based on a 3D medical world model.

Figure 2. Overview of the proposed framework. The 3D medical world model framework consisting of a perception-simulation-decision cycle composed of 3D MedGemma, 3D Conditional Diffusion Model, and Feasibility Guided Plan Selection.

The framework consists of three core components:

  1. Perceive (3D MedGemma): The authors extended the 2D MedGemma vision-language model into 3D. By replacing the vision encoder with a 3D SigLIP-based counterpart, the model can process raw, volumetric CT scans alongside textual prompts to propose context-aware, natural-language surgical actions.
  2. Simulate (3D Conditional Diffusion Model): Acting as the "world model," this component predicts the physical consequences of a proposed surgical action. Built on a Denoising Diffusion Probabilistic Model (DDPM), it generates high-fidelity 3D segmentation masks showing exactly which anatomical structures will be affected or resected.
  3. Decide (Feasibility Filtering & Global Optimization): A two-stage decision engine evaluates the proposed steps. First, a Local Feasibility Check filters out physically impossible or dangerous actions based on depth, occlusion, accessibility, and surgical sequencing constraints (e.g., you cannot resect a child bronchus before its parent). Second, a Global Optimizer uses beam search to select the path that maximizes tumor clearance while minimizing the loss of functional lung volume.

Datasets

The models were trained and evaluated on two newly constructed clinical datasets developed in collaboration with thoracic surgeons:

  • Chest CT-Report Dataset: Contains 1,000 paired chest CT scans and preoperative radiology reports. This dataset was used for the contrastive pre-training of the 3D Vision Encoder to align visual features with medical text.
  • Surgical History Dataset: Comprises 200 clinical cases of anatomical segmentectomies or subsegmentectomies. Each case includes a preoperative CT scan and detailed surgical notes. The broncho-vascular structures were automatically segmented using an nnU-Net model and aligned with surgical actions to create action-mask pairs for training the diffusion simulator.

Key Insights and Ideas

  • Anatomy as a Dynamic Environment: Rather than treating medical imaging as a static map, this work treats the patient's body as a dynamic environment where actions have consequences. This shifts surgical AI from simple classification/segmentation toward true causal reasoning.
  • Action-Conditioned 3D Simulation: Using diffusion models to generate 3D masks of what will change given a specific surgical action provides a highly interpretable, visual feedback loop for the AI planner.
  • Synergy of LLMs and Geometric Constraints: Pure language models lack spatial awareness, and pure geometric planners lack clinical reasoning. Combining 3D MedGemma with a discrete feasibility/safety filter bridges the gap between high-level language planning and low-level anatomical reality.

Key Results

The proposed framework significantly outperformed all baseline methods (including rule-based planners, reinforcement learning planners, and PlanLLM) across all quantitative metrics:

  • Step Accuracy (SA): Achieved 71.2%, a substantial improvement over the best baseline (MedGemma+RAG at 59.4%).
  • Critical Step Accuracy (CSA): Reached 84.3%, proving the model's ability to preserve the exact, clinically essential order of critical operations.
  • Semantic Similarity: Scored 0.92, indicating the generated plans closely matched the natural language style of expert surgeons.

A blind user study conducted with thoracic surgeons confirmed these findings. The surgeons rated the proposed model's plans significantly higher than the baseline across clinical reasonableness, anatomical feasibility, and lung preservation, expressing strong overall preference for the world-model approach.

Limitations and Future Work

While promising, the authors note a few key limitations:

  • Anatomical Outliers: The generative components can still struggle when encountering extreme or highly rare anatomical variations.
  • Inference Speed: Because the 3D diffusion simulator relies on iterative denoising steps, real-time deployment during active surgery is currently limited.
  • Sample Size: The user study, while qualitatively positive, only involved two thoracic surgeons and requires broader clinical validation.

Summary updated

Abstract

Automated planning for segmental/subsegmental pulmonary resection remains challenging due to three key limitations: (1) static anatomical perception that fails to capture patient-specific broncho-vascular complexity; (2) absence of preoperative simulation that predicts how each surgical action alters anatomy; and (3) weak integration of clinical reasoning into end-to-end decision-making. Prior methods, ranging from rule-based planners to large language models and reinforcement learning, fail to model 3D anatomy, forecast anatomical outcomes, or reliably produce feasible surgical sequences. To overcome these gaps, we propose a 3D medical world model for pulmonary surgery with a closed-loop "Perceive-Simulate-Decide" architecture. Our framework introduces three innovations: First, 3D MedGemma, a vision-language model that interprets CT and text to generate anatomy-aware actions. Second, a 3D conditional diffusion simulator that predicts action-induced tissue changes via precise segmentation. Third, a clinically grounded decision engine that enforces stepwise feasibility and globally optimizes tumor clearance against parenchymal preservation. On clinical cases, our method achieves SOTA results: step accuracy 71.2%, critical step accuracy 84.3%, overlap 78.9%, semantic similarity 0.92, outperforming all baselines. Surgeons consistently rate our plans as more feasible and lung-preserving. To our knowledge, this is the first end-to-end inference framework capable of generating surgical plans directly from 3D imaging, establishing a new paradigm for procedural AI in surgery. Code and data will be released.
BibTeX
@inproceedings{zhang2026surgical,
  author = {Zhen Zhang and Zhaorong Dong and Xiao Yang and Liqin Huang and Qiang Wu and Taidui Zeng and Hanyu Zheng and Mingjing Yang and Shaohua Zheng and Wangbin Ding and Lin Pan},
  title = {{Surgical Procedural Planning as 3D World Modelling: Towards Automated Pulmonary Resection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=jTeTsyTyKY},
}
Figure