Introduction
Neurosurgery requires extreme precision, often involving the navigation of complex 3D structures based on 2D Magnetic Resonance (MR) images. This discrepancy creates a significant cognitive load, as surgeons must mentally translate 2D slices into a 3D spatial map while operating. Traditional training methods are often limited by cost and the lack of realistic physical feedback. This paper presents a high-fidelity neurosurgical training simulator that integrates patient-specific 3D brain models with Augmented Reality (AR) and bimanual haptic feedback to bridge the gap between imaging and reality.
The Digital Twin Approach
The proposed system creates a "digital twin" of the surgical procedure, allowing trainees to interact with a virtual brain overlaid on a physical 3D-printed skull. The workflow encompasses anatomy reconstruction, spatial registration, biomechanical modeling, and haptic interaction.
Anatomy Reconstruction and Biomechanical Modeling
The process begins with patient-specific MRI data. A deep learning framework, VoxResNet, is utilized to segment brain tissue, vessels, and tumors, which are then converted into 3D polygon meshes. To simulate the physical properties of brain tissue, the system employs a mass-spring model. This allows for realistic deformation and "snapping" (cutting) of the virtual tissue when surgical tools apply force.
Virtual-Real Spatial Registration
A critical component of the simulator is ensuring the hologram aligns perfectly with the physical environment. The team developed a four-step registration process using a Microsoft HoloLens for visualization and an NDI stereo tracking system for spatial calibration. This setup ensures that the virtual brain is accurately positioned within a 3D-printed physical skull template.
Bimanual Haptic Interaction
Unlike many simulators that use a single controller, this system integrates two Geomagic Touch devices to facilitate realistic two-handed surgery. A multirate technique is used to synchronize visual and haptic feedback: while visual rendering runs at 25–60 Hz, the haptic loop operates at 1000 Hz. This ensures that the force feedback feels smooth and responsive, even during complex maneuvers like tissue retraction and tumor resection.
Results and Validation
The system's performance was evaluated based on spatial accuracy and user feedback.
- Accuracy: The system achieved an average Target Registration Error (TRE) of 2.11 mm, meeting the precision requirements for clinical training.
- User Validation: In a study of 10 participants, the simulator received high marks for realism. 90% of users found the AR visualization realistic, and 80% agreed the haptic feedback was lifelike. Furthermore, 90% of participants considered the system a sufficient tool for training neurosurgical skills.
Conclusion and Future Directions
The integration of AR and bimanual haptics provides a promising "see-through" training mode that maintains the context of the real surgical field. While the current model lacks dynamic details like blood flow and complex boundary conditions, it represents a significant step toward high-fidelity, patient-specific neurosurgical simulation. Future work will focus on improving dynamic realism and transitioning the registration pipeline to live clinical scenarios.
Summary updated