Development and Pre-Clinical Analysis of Spatiotemporal-Aware Augmented Reality in Orthopedic Interventions

Javad Fotouhi, Arian Mehrfard, Tianyu Song, Alex Johnson, Greg Osgood, Mathias Unberath, Mehran Armand, Nassir NavabIEEE TMI 2021 · DOI

TL;DR

This paper introduces a novel spatiotemporal-aware augmented reality (AR) system designed to improve orthopedic surgical interventions. By leveraging the "viewing frustum" of imaging devices and users, the system provides intuitive, co-registered 3D visualizations of patient anatomy and surgical tools, leading to reduced procedure times, fewer X-ray acquisitions, and lower radiation doses compared to traditional methods.

What Problem Does This Paper Address?

Image-guided orthopedic interventions, while crucial for minimally invasive procedures, often suffer from suboptimal interaction with patient data and challenges in understanding 3D anatomy from 2D X-ray images. This can lead to inefficient workflows, increased radiation exposure, and potential complications. Existing AR solutions have primarily focused on visualization or basic overlays, but this paper tackles the need for a more integrated, interactive, and collaborative AR experience that redefines surgical workflows by providing real-time, spatially aware guidance.

Methods

The core of the proposed system is a "spatiotemporal-aware AR" concept that utilizes the concept of "flying frustums." This approach models the viewable region of both the X-ray imaging device and the human observers (surgeons) in the operating room. Key methodological contributions include:

  • Spatial-Awareness for AR: Employing Simultaneous Localization and Mapping (SLAM) to enable marker-free co-localization of augmented users and the imaging device, allowing 3D information to propagate seamlessly.
  • Imaging Observer Integration: Co-calibrating the C-arm scanner with a visual tracker to accurately determine its pose in the operating room environment.
  • Geometry-Awareness for AR: Explicitly modeling the flying frustum of the X-ray camera to allow interaction with images within their geometries, enabling intuitive 2D-image-to-3D-anatomy associations.
  • Planning Using Flying Frustums: Two approaches for intra-operative planning are presented:
  • Forward-projection of virtual implants: Projecting 3D virtual surgical implants onto X-ray images for pre-placement verification.
  • Multi-view geometry for landmark reconstruction: Using intersections of rays from multiple views to reconstruct 3D landmarks and trajectories.
  • Surgical Workflow Integration: Demonstrating how the system can be integrated into procedures like Total Hip Arthroplasty (THA) and K-wire placement for fracture repair.

Datasets

Not explicitly discussed in the paper as distinct datasets for training. The system is evaluated using pre-clinical studies involving orthopedic surgeons performing specific tasks on phantoms.

Key Insights and Ideas

  • Leveraging the "Viewing Frustum": The paper's central innovation is using the concept of viewing frustums (both for imaging devices and users) to create a spatially aware AR environment. This allows for a more intuitive and context-aware visualization of surgical information.
  • Team-Centric AR: The system is designed to facilitate collaboration, allowing multiple users (surgeon, crew, technician) to share the same AR experience and information, improving communication.
  • Reducing Reliance on External Tracking: By using marker-free SLAM and co-calibration, the system aims to minimize the need for cumbersome external tracking hardware, simplifying setup and workflow.
  • Bridging 2D and 3D: The system effectively bridges the gap between 2D X-ray images and 3D patient anatomy, enabling more accurate planning and execution.
  • Workflow Integration, Not Replacement: The AR system is designed to augment, not replace, the standard surgical workflow, making it more translatable and safer.

Key Results

The study demonstrated significant improvements in pre-clinical orthopedic procedures:

  • K-wire Insertion:
    • Time: AR procedure took on average 111.25 sec, compared to 594.3 sec for Standard Operating Procedure (SOP).
  • Radiation Dose: Significantly lower dose with AR (0.255 cGY(cm²)) compared to SOP (4.43 cGY(cm²)).
  • X-ray Acquisitions: Reduced to 2 images per procedure with AR, compared to an average of 40.86 for SOP.
  • Accuracy: AR achieved an insertion error of 4.76 mm, comparable to SOP (4.61 mm), with no statistically significant difference.
  • Acetabular Cup Placement (THA):
    • Time: AR procedure took on average 122.38 sec, compared to 235 sec for SOP.
  • Radiation Dose: Lower dose with AR (1.25 cGY(cm²)) compared to SOP (1.96 cGY(cm²)).
  • X-ray Acquisitions: Reduced to 8 images per procedure with AR, compared to 13.75 for SOP.
  • Accuracy: AR resulted in significantly lower abduction (1.57° vs. 4.76°) and anteversion (1.46° vs. 4.78°) errors compared to SOP.

Limitations and Future Work

  • Limitations:
    • The most significant source of error is identified as the localization of the AR HMD using SLAM, which can lead to drifts.
  • The current system's accuracy, while comparable to SOP in some aspects, may not yet match marker-based surgical navigation or robotic systems.
  • The study was pre-clinical, using phantoms.
  • Future Work:
    • Integrating additional sensors (IMUs, depth cameras) to improve SLAM tracking accuracy and reduce drifts.
  • Further development of computer vision methodologies to enhance accuracy and robustness.
  • Exploring the integration of AI for semantic understanding of the surgical environment and augmenting surgeon intelligence.
  • Investigating the use of eye-tracking systems for improved calibration and interaction.
  • Expanding the application to other surgical fields and developing "surgical replay" functionalities for training.
  • Addressing potential perceptual ambiguities (vergence-accommodation conflict) and exploring haptic feedback.

Summary updated

BibTeX
@article{fotouhi2021development,
  author = {Javad Fotouhi and Arian Mehrfard and Tianyu Song and Alex Johnson and Greg Osgood and Mathias Unberath and Mehran Armand and Nassir Navab},
  title = {{Development and Pre-Clinical Analysis of Spatiotemporal-Aware Augmented Reality in Orthopedic Interventions}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2021},
  doi = {10.1109/tmi.2020.3037013},
}
Figure