TL;DR
This paper introduces the Stereoscopic AR Predictive Display (SARPD), the first system to use augmented reality to predict and display a surgical robot's movements in real-time. SARPD significantly reduces task completion time in telesurgery under delay without increasing errors, making remote surgery more efficient and safer.
What Problem Does This Paper Address?
Remote surgery, or telesurgery, holds immense promise for extending surgical expertise to underserved areas and enabling complex procedures. However, a major hurdle is the inherent latency between the surgeon's commands and the robot's actions. This delay, caused by the time it takes for signals to travel, can lead to significant inaccuracies, overshoot, and oscillations, making precise surgical maneuvers difficult and potentially dangerous. While predictive displays have shown success in other fields by offering operators a glimpse of future states, applying them to telesurgery is challenging due to the need for highly accurate 3D tracking of the surgical environment and the non-rigid nature of tissue. This paper aims to bridge this gap by developing and evaluating a novel AR-based predictive display specifically for teleoperated surgical robots.
Methods
The core of the SARPD system lies in its ability to accurately track the surgical robot's tools and render a predictive AR overlay. The system employs an Extended Kalman Filter (EKF) to continuously correct the hand-eye transform between the robot's base and the camera. This EKF-based approach addresses challenges arising from imperfect kinematic calibrations and allows for on-the-fly adjustments.
For the visual display, SARPD utilizes Augmented Reality (AR) to overlay the predicted movements of the robot's slave tools onto the real-time camera feed. This AR rendering is stereoscopic, providing depth perception for the surgeon. A key innovation is the use of dynamic transparency, where the opacity of the predicted AR overlay adjusts based on the predicted tool's distance from its target. This ensures that the AR prediction is most visible when needed for larger movements and becomes more transparent during precise, slow maneuvers where visual feedback is paramount. The system is built using The Visualization Toolkit (VTK) and OpenGL, and runs on a consumer-grade GPU.
Datasets
The paper does not explicitly mention using pre-existing datasets for training. Instead, it relies on:
- Real-time data captured from the da Vinci Surgical System during user studies.
- 3D CAD models of the surgical tools for AR rendering.
- Checkerboard patterns used for initial hand-eye calibration.
The user study itself serves as the primary evaluation benchmark, comparing performance with and without SARPD under delayed conditions.
Key Insights and Ideas
- AR as a Natural Fit for Surgical Prediction: Unlike previous predictive displays that might require assumptions about the environment, AR can directly overlay predictions onto the real-world view, making it highly intuitive for surgeons.
- Dynamic Transparency is Crucial: The adaptive opacity of the AR overlay is a clever solution to balance the need for predictive guidance with the importance of direct visual feedback during delicate maneuvers. It prevents visual clutter and confusion.
- EKF for Robust Tool Tracking: The use of an EKF to continuously refine the hand-eye calibration is essential for maintaining accurate predictions, especially in dynamic surgical environments where minor inaccuracies can be amplified.
- Real-time Performance on Consumer Hardware: Achieving high frame rates for both AR rendering and tool tracking on a standard GPU is a significant engineering feat, making the technology more accessible.
Key Results
The user study, conducted with ten participants (seven novices, three surgeons) on the da Vinci Surgical System, yielded significant findings:
- Reduced Task Completion Time: SARPD demonstrated a statistically significant decrease in the time required to complete tasks when operating under a 1-second round-trip delay. On average, task completion time was reduced by 19%.
- No Increase in Errors: Crucially, the use of SARPD did not lead to a statistically significant increase in weighted errors. This indicates that the predictive display improves efficiency without compromising safety.
- Efficient System Performance: The SARPD system achieved real-time slave-tool tracking at 24Hz and stereoscopic AR rendering at over 30fps. The overall image pipeline latency was measured at 100±20ms, an improvement over standard ROS image viewing.
Limitations and Future Work
The authors acknowledge several limitations and suggest avenues for future research:
- Limited Task Scope: The user study focused on a peg transfer task. Future work should explore a wider range of surgical tasks, particularly those involving more complex environmental interactions.
- Cognitive Load: More studies are needed to rigorously assess the cognitive load on surgeons when operating with and without SARPD under various delay conditions.
- Endoscopic Motion Tracking: The current tracking algorithm needs to be extended to incorporate endoscopic motions, which are a standard part of surgical procedures.
- Obstruction Handling: The rendering pipeline should be enhanced to account for potential obstructions of the slave tools by environmental obstacles.
- Subjective Data: Gathering more subjective data from users would provide deeper insights into their experience and the perceived benefits of SARPD.
Summary updated