Twin-S: a digital twin for skull base surgery

Hongchao Shu, Ruixing Liang, Zhaoshuo Li, Anna Goodridge, Xiangyu Zhang, Hao Ding, Nimesh Nagururu, Manish Sahu, Francis X. Creighton, Russell H. Taylor, Adnan Munawar, Mathias Unberath

TL;DR

Twin-S is a real-time digital twin framework designed to provide surgeons with a high-fidelity virtual replica of skull base surgery as it happens. By combining high-precision optical tracking with physics-based simulation, the system updates the patient’s anatomical model in real-time as bone is drilled away, enabling advanced intra-operative guidance and situational awareness.

What Problem Does This Paper Address?

Skull base surgeries, such as mastoidectomies, are high-stakes procedures where surgeons must navigate a complex "minefield" of cortical bone to reach the middle and inner ear. Critical structures—including nerves, major blood vessels, and end-organs—are often buried within the bone, invisible to the naked eye.

While Augmented Reality (AR) and Virtual Reality (VR) have been introduced to help, they have significant gaps. Most current AR systems overlay static pre-operative scans that don't reflect the physical changes (like tissue removal) occurring during the operation. Conversely, VR systems are great for simulation but typically don't "listen" to real-world measurements. Twin-S bridges this gap by creating a "live" digital counterpart that evolves alongside the physical surgery.

Methods

The Twin-S paradigm relies on a continuous loop between the Physical Twin (the actual surgical setup) and the Digital Twin (the simulation).

  1. Tracking & Modeling: The system uses an optical tracker (Atracsys FusionTrack 500) to monitor the 3D poses of the surgical drill, the patient (a surgical phantom), and the stereo microscope.
  2. Physics-Based Simulation: These poses are streamed into the Asynchronous Multi-Body Framework (AMBF). The digital twin uses a pre-operative CT scan of the phantom to create a voxel-based occupancy map.
  3. Real-Time Interaction: As the surgeon drills in the real world, the system detects collisions between the virtual drill tip (modeled as a sphere) and the virtual bone. It removes the corresponding voxels in real-time, updating the anatomical model at a frame rate of 28 FPS.
  4. Calibration: To ensure the virtual world matches the real one, the team implemented rigorous calibration routines, including pivot calibration for the drill, hand-eye calibration for the camera, and Iterative Closest Point (ICP) registration to align the virtual CT model with the physical phantom.

Datasets

The study utilized a specialized surgical phantom to simulate human anatomy.

  • Anatomical Data: A high-resolution CT scan of the phantom (using a Brainlab Loop-X scanner) served as the baseline for the digital twin.
  • Evaluation Data: To validate the system, a surgical expert performed a simulated mastoidectomy. The "post-op" phantom was then CT-scanned again to provide a ground truth for the bone removed.
  • Labeling: For the segmentation mask evaluation, the authors manually labeled a sequence of 100 video frames to compare real-world tool/anatomy positions against the virtual generated masks.

Key Insights and Ideas

  • Closing the Feedback Loop: The most significant insight is the transition from static overlays to a dynamic model. By updating the anatomy in real-time, the digital twin can provide "temporally varying" feedback—essentially a live progress report of the surgery.
  • The "Safety Buffer" Visualization: The authors demonstrated a mixed-reality use case where the system provides a color-coded overlay. As the surgeon drills closer to a pre-planned target (or a critical structure), the overlay shifts from "warm" to "cool" colors, providing intuitive depth cues that are otherwise hard to perceive through a microscope.
  • Error Decomposition: The paper provides a detailed numerical analysis of where errors come from. They found that while optical tracking is incredibly precise (~0.02 mm), the cumulative error from calibration (drill and camera) is the primary bottleneck, reaching a theoretical worst-case of 1.7° in rotation.

Key Results

  • Drilling Accuracy: The system achieved an average error of 1.39 mm in the drilled regions when compared to the post-operative ground truth CT scan. This is comparable to the precision of conventional optical navigation systems currently used in hospitals.
  • Latency: The total computational pipeline takes roughly 35.7 ms, allowing the system to run at 28 FPS, which is sufficient for real-time visual feedback.
  • Segmentation Fidelity: The system generated virtual segmentation masks with high accuracy, achieving a Dice score of 0.956 for the patient anatomy and 0.725 for the surgical drill.

Limitations and Future Work

The primary limitation identified is the line-of-sight issue inherent to optical tracking; if a sensor is blocked, the twin loses its "vision." Consequently, the drilling depth in this study was shallower than in actual clinical practice to maintain a clear line of sight.

The authors suggest that future iterations should integrate vision-based tracking (using the microscope's video feed) to augment optical tracking. Furthermore, they envision Twin-S as a massive data engine: by pairing real microscopic video with virtually generated labels, they can create large-scale datasets to train AI models for surgery without the need for tedious manual labeling.

Summary updated

Abstract

Digital twins are virtual replicas of real-world objects and processes, and they have potential applications in the field of surgical procedures, such as enhancing situational awareness. We introduce Twin-S, a digital twin framework designed specifically for skull base surgeries.
BibTeX
@article{shu2023twin,
  author = {Hongchao Shu and Ruixing Liang and Zhaoshuo Li and Anna Goodridge and Xiangyu Zhang and Hao Ding and Nimesh Nagururu and Manish Sahu and Francis X. Creighton and Russell H. Taylor and Adnan Munawar and Mathias Unberath},
  title = {{Twin-S: a digital twin for skull base surgery}},
  journal = {Int. J. Comput. Assist. Radiol. Surg.},
  year = {2023},
  url = {https://openreview.net/forum?id=hdlbksH0rU},
}

Loading related content...

Figure