Abstract
This work presents a novel method for augmented reality in liver surgery. Leveraging advanced reconstruction and segmentation techniques, our approach transfers diverse intraoperative data to Euclidean space, using vessel centerlines as a common representation. Hence, overcoming limitations in operation room equipment and clinician expertise. In addition, we capitalize on the graph-like structure of intraoperative vascular trees to ensure a robust and automatic initial registration. Our approach combines physics-based modeling accuracy with neural network speed for real-time elastic registration. The proposed method is trained on data synthesized from real human acquisitions and validated against a biomechanical model. Our approach achieves a mean target registration error less than 0.6 mm for different synthesized intraoperative cases. Moreover, it demonstrates robustness over different modalities and segmentation errors.
Introduction
During abdominal surgery, the liver undergoes significant deformation due to factors such as respiration, surgical manipulation, and changes in pneumoperitoneum pressure. To provide effective surgical guidance via Augmented Reality (AR)—overlaying 3D models of internal tumors and vascular structures onto the surgeon's live view—a system must accurately align (register) pre-operative 3D scans with the deformed intra-operative state in real-time.
Traditional registration methods face a difficult trade-off: physics-based biomechanical models offer high accuracy but are computationally expensive and slow, while deep learning methods provide speed but are often constrained to specific imaging modalities (e.g., CT-to-Ultrasound). Furthermore, many existing techniques rely on organ surface mapping, which requires complex and often manual initial alignment before tracking deformations.
Proposed Methodology
The authors propose a patient-specific pipeline that bridges the gap between physics-based accuracy and deep learning speed. The core of this approach is the use of internal blood vessel centerlines as a universal, modality-agnostic landmark.
1. Graph-Based Initial Registration
Rather than relying on the liver's surface, the system utilizes the internal vascular tree (the portal vein). By extracting vessel centerlines, the network is treated as a mathematical graph. The system automatically identifies the root branch to robustly snap pre-operative and intra-operative models into initial alignment using an Iterative Closest Point (ICP) algorithm.
2. Physics-Based Digital Twin
To overcome the lack of large-scale intra-operative datasets, the authors built a biomechanical Finite Element Method (FEM) model of the liver. This "digital twin" was used to simulate 3,600 unique surgical deformations by applying random forces to the liver surface, calculating the precise displacement of internal structures to serve as ground-truth training data.
3. Real-Time Non-Rigid Registration
A U-Net neural network is trained on the simulated data. It takes a 3D grid encoding the intra-operative vessel centerlines as input and predicts the non-linear displacement field required to warp the pre-operative model to the patient's current anatomy. This allows the system to remain modality-agnostic; as long as vessels can be segmented from Ultrasound, CBCT, or MRI, the registration can proceed.
Key Results and Performance
The method was validated using synthetic datasets derived from real patient anatomy (3D-IRCADb database), including "partial view" scenarios to mimic the limited field of view of intra-operative ultrasound.
- Accuracy: The system achieved a mean Target Registration Error (TRE) of 0.60 mm with a full vessel tree and maintained high accuracy (approx. 0.63 mm) even when only 50% of the vessel tree was visible.
- Speed: The neural network computes complex non-rigid deformations in just 34 milliseconds, enabling smooth, real-time AR overlays.
- Robustness: The pipeline demonstrated high tolerance to noise, maintaining performance even with up to 2 mm of artificial error injected into the centerline data to simulate imperfect segmentation.
Conclusion
By combining the structural insights of vascular graphs with the predictive power of deep learning trained on physics-based simulations, this research provides a viable path toward real-time, modality-independent AR in the operating room. While currently validated on synthetic deformations, future work will focus on testing the digital twin against real ex-vivo human liver data to further refine the model's boundary conditions and physical accuracy.
Summary updated