HoloLens augmented reality system for transperineal free-hand prostate procedures

Ming Li, Sherif Mehralivand, Sheng Xu, Nicole Varble, Ivane Bakhutashvili, Sandeep Gurram, Peter A. Pinto, Peter L. Choyke, Bradford J. Wood, Barış TürkbeyJournal of Medical Imaging 2023 · DOI

TL;DR

Researchers have developed a HoloLens-based augmented reality (AR) system to improve the accuracy and flexibility of transperineal (TP) prostate procedures. This system overlays 3D anatomical information onto the patient in real-time, offering a more intuitive and immersive guidance experience compared to traditional methods, potentially leading to better patient outcomes.

What Problem Does This Paper Address?

Transperineal (TP) prostate procedures, such as biopsies and treatments like focal laser ablation or brachytherapy, are crucial for diagnosing and treating prostate cancer. However, current guidance methods for these procedures, particularly free-hand techniques, face limitations. Traditional grid-based guidance systems restrict needle trajectories to a fixed parallel path, which can hinder access to certain lesions, especially in larger prostates or those near anatomical barriers like the pubic bone. While some advanced methods like MRI-ultrasound fusion exist, they can be complex or have their own drawbacks. The challenge is to provide real-time, flexible, and accurate needle guidance for TP procedures that overcomes the limitations of existing methods, enhancing both precision and ease of use for clinicians.

Methods

The core of this work is a HoloLens-based AR system designed for transperineal prostate procedures. The system integrates preprocedural volumetric imaging data (MRI/CT) with the patient's real-world anatomy. Key components include:

  • Patient Reference Frame: A custom-designed frame attached to the patient, featuring fiducial markers. This frame acts as a crucial anchor for registering the digital AR content with the physical patient.
  • AR Planning Software: Used for segmenting anatomical features from medical images, planning the needle trajectory, and generating 3D models.
  • HoloLens Application: This novel application overlays the planned needle path, target location, and skin entry point onto the patient's view through the HoloLens headset. It also provides real-time visualization of the needle's progress.
  • Needle Tracking: A needle reference frame is attached to the biopsy needle. The HoloLens' camera tracks this frame, allowing for real-time visualization of the digital needle overlaid on the actual needle.

Two guidance modes were implemented and evaluated:

  • Planned-Path Guidance: Displays the planned needle trajectory, target, and entry point. The user aligns the actual needle with the digital path.
  • Free-Hand Guidance: Primarily displays the digital target. The needle is tracked in real-time, and its position and angle are visualized, offering more flexibility in choosing the entry point.

Datasets

The study utilized phantoms for evaluation, rather than patient datasets for training.

  • 3D-Printed Phantom: Used for initial system accuracy evaluation.
  • Gel Phantom: Employed for evaluating needle placement accuracy and time.
  • Anthropomorphic Pelvic Phantom (CIRS Model 048A): This phantom, containing simulated tumors (pseudolesions), was used to assess the feasibility of the HoloLens AR guidance in a more realistic scenario for seed implantation.

Key Insights and Ideas

  • Immersive 3D Guidance: The HoloLens' ability to display a true 3D hologram of the digital object within the real-world environment offers a significantly more intuitive and immersive experience than traditional 2D monitors. This "hands-free" approach can increase spatial awareness during procedures.
  • Flexibility of Free-Hand AR: The free-hand guidance mode, while requiring more user training and careful tracking of both the patient frame and needle frame, offers greater flexibility in choosing needle entry points compared to rigid grid systems.
  • Bridging the Gap: The AR system effectively bridges the gap between preprocedural imaging and real-time procedural guidance, addressing the challenge of visualizing internal anatomy and needle position in free-hand TP procedures.
  • Potential for Reduced Sepsis Risk: By potentially improving the accuracy and control of TP biopsies, the system could further solidify the TP approach's advantage of a lower sepsis risk compared to transrectal biopsies.

Key Results

  • Image Overlay Accuracy: The system achieved an image overlay error of 1.29 ± 0.57 mm.
  • Visual Needle Targeting Error: The error in visually targeting the needle was 2.13 ± 0.52 mm.
  • Needle Placement Error: Both planned-path and free-hand guidance methods resulted in similar needle placement errors: 4.14 ± 1.08 mm for planned-path and 4.20 ± 1.08 mm for free-hand (p = 0.90).
  • Needle Placement Time: The AR-guided planned-path method was slightly faster (47 ± 22 s) compared to the AR-guided free-hand method (65 ± 19 s).
  • Feasibility: The system demonstrated feasibility in a seed implantation experiment, with markers successfully implanted into or near target lesions.

Limitations and Future Work

  • System Error vs. Operator Error: The observed needle placement error is a combination of system accuracy and subjective operator error. While system errors were low, operator training and experience with the HoloLens significantly influence performance.
  • Tracking Limitations: The current system relies on keeping image patterns (patient reference frame and needle reference frame) within the HoloLens camera's field of view. This can be challenging, and slight shifts in needle alignment were observed when users moved their heads.
  • Lack of Real-Time Motion Correction: The system is based on preprocedural 3D images and cannot automatically correct for organ motion or deformation without additional real-time imaging modalities like ultrasound (US).
  • Future Directions: The authors suggest combining real-time US with the AR system for improved accuracy, especially for procedures involving organ motion. Further investigation into the impact of operator training and experience is also warranted. The potential for this AR system to address unmet clinical needs in TP procedures, particularly when needle visualization is challenging, is highlighted.

Summary updated

Abstract

The HoloLens AR system can provide accurate needle guidance for TP interventions. AR support for free-hand lesion targeting is feasible and may provide more flexibility than grid-based methods, due to the real-time 3D and immersive experience during free-hand TP procedures.
BibTeX
@article{li2023hololens,
  author = {Ming Li and Sherif Mehralivand and Sheng Xu and Nicole Varble and Ivane Bakhutashvili and Sandeep Gurram and Peter A. Pinto and Peter L. Choyke and Bradford J. Wood and Barış Türkbey},
  title = {{HoloLens augmented reality system for transperineal free-hand prostate procedures}},
  journal = {Journal of Medical Imaging},
  year = {2023},
  doi = {10.1117/1.jmi.10.2.025001},
}
Figure