Assessing performance of augmented reality-based neurosurgical training

Wei-Xin Si, Xiang-Yun Liao, Yin-Ling Qian, Hai-Tao Sun, Xiang-Dong Chen, Qiong Wang, Pheng-Ann Heng

Introduction

Neurosurgery requires extreme precision, often involving the navigation of complex 3D structures based on 2D Magnetic Resonance (MR) images. This discrepancy creates a significant cognitive load, as surgeons must mentally translate 2D slices into a 3D spatial map while operating. Traditional training methods are often limited by cost and the lack of realistic physical feedback. This paper presents a high-fidelity neurosurgical training simulator that integrates patient-specific 3D brain models with Augmented Reality (AR) and bimanual haptic feedback to bridge the gap between imaging and reality.

The Digital Twin Approach

The proposed system creates a "digital twin" of the surgical procedure, allowing trainees to interact with a virtual brain overlaid on a physical 3D-printed skull. The workflow encompasses anatomy reconstruction, spatial registration, biomechanical modeling, and haptic interaction.

Overview of the AR-based neurosurgery training system, showing the pipeline from MRI scanning to bimanual haptic interaction.

Anatomy Reconstruction and Biomechanical Modeling

The process begins with patient-specific MRI data. A deep learning framework, VoxResNet, is utilized to segment brain tissue, vessels, and tumors, which are then converted into 3D polygon meshes. To simulate the physical properties of brain tissue, the system employs a mass-spring model. This allows for realistic deformation and "snapping" (cutting) of the virtual tissue when surgical tools apply force.

Virtual-Real Spatial Registration

A critical component of the simulator is ensuring the hologram aligns perfectly with the physical environment. The team developed a four-step registration process using a Microsoft HoloLens for visualization and an NDI stereo tracking system for spatial calibration. This setup ensures that the virtual brain is accurately positioned within a 3D-printed physical skull template.

Detailed view of the virtual-real spatial information visualization registration, including stereo tracking and HoloLens calibration.

Bimanual Haptic Interaction

Unlike many simulators that use a single controller, this system integrates two Geomagic Touch devices to facilitate realistic two-handed surgery. A multirate technique is used to synchronize visual and haptic feedback: while visual rendering runs at 25–60 Hz, the haptic loop operates at 1000 Hz. This ensures that the force feedback feels smooth and responsive, even during complex maneuvers like tissue retraction and tumor resection.

Results and Validation

The system's performance was evaluated based on spatial accuracy and user feedback.

  • Accuracy: The system achieved an average Target Registration Error (TRE) of 2.11 mm, meeting the precision requirements for clinical training.
  • User Validation: In a study of 10 participants, the simulator received high marks for realism. 90% of users found the AR visualization realistic, and 80% agreed the haptic feedback was lifelike. Furthermore, 90% of participants considered the system a sufficient tool for training neurosurgical skills.

Conclusion and Future Directions

The integration of AR and bimanual haptics provides a promising "see-through" training mode that maintains the context of the real surgical field. While the current model lacks dynamic details like blood flow and complex boundary conditions, it represents a significant step toward high-fidelity, patient-specific neurosurgical simulation. Future work will focus on improving dynamic realism and transitioning the registration pipeline to live clinical scenarios.

Summary updated

Abstract

This paper presents a novel augmented reality (AR)-based neurosurgical training simulator which provides a very natural way for surgeons to learn neurosurgical skills. Surgical simulation with bimanual haptic interaction is integrated in this work to provide a simulated environment for users to achieve holographic guidance for pre-operative training. To achieve the AR guidance, the simulator should precisely overlay the 3D anatomical information of the hidden target organs in the patients in real surgery. In this regard, the patient-specific anatomy structures are reconstructed from segmented brain magnetic resonance imaging. We propose a registration method for precise mapping of the virtual and real information. In addition, the simulator provides bimanual haptic interaction in a holographic environment to mimic real brain tumor resection. In this study, we conduct AR-based guidance validation and a user study on the developed simulator, which demonstrate the high accuracy of our AR-based neurosurgery simulator, as well as the AR guidance mode’s potential to improve neurosurgery by simplifying the operation, reducing the difficulty of the operation, shortening the operation time, and increasing the precision of the operation.
BibTeX
@article{si2019assessing,
  author = {Wei-Xin Si and Xiang-Yun Liao and Yin-Ling Qian and Hai-Tao Sun and Xiang-Dong Chen and Qiong Wang and Pheng-Ann Heng},
  title = {{Assessing performance of augmented reality-based neurosurgical training}},
  journal = {Vis. Comput. Ind. Biomed. Art},
  year = {2019},
  doi = {10.1186/s42492-019-0015-8},
}
Figure