Reflective-AR Display: An Interaction Methodology for Virtual-Real Alignment in Medical Robotics

Javad Fotouhi, Tianyu Song, Arian Mehrfard, Giacomo Taylor, Alejandro Martin-Gomez, Bernhard Fuerst, Mehran Armand, Mathias Unberath, Nassir NavabarXiv 2019 · arXiv

Abstract

In robot-assisted minimally invasive surgery, the manual setup and positioning of robotic arms represent a significant operational bottleneck. Sub-optimal placement often leads to intraoperative collisions or restricted range of motion, necessitating time-consuming resets. While Augmented Reality (AR) offers a potential solution for guidance, "perspective ambiguity"—the difficulty in perceiving depth and alignment through a single viewpoint—limits its effectiveness. This paper introduces Reflective-AR, a methodology that utilizes virtual mirrors to provide simultaneous multi-perspective views, and ViRAAl (Virtual-Real Active Alignment), a registration algorithm using Riemannian averaging. Experimental results demonstrate that this approach reduces alignment errors by nearly 63% compared to standard AR methods.

Introduction

The "setup" phase of robotic surgery involves the precise positioning and docking of robotic arms relative to the patient. Due to stringent safety regulations, this process remains largely manual. If the arms are positioned poorly, they may collide with one another or fail to reach the surgical site, resulting in "junk time" where the surgery must be paused to undock and reset the equipment.

Augmented Reality (AR) has been proposed to streamline this process by overlaying a "ghost" model of the ideal robot configuration. However, human depth perception in AR is notoriously unreliable. When a user views a virtual object from a single perspective, it is difficult to determine if the virtual model is truly aligned with the physical robot or if it merely appears aligned due to the user's specific viewpoint. This phenomenon, known as perspective ambiguity, remains a primary hurdle for AR-guided medical interventions.

The Reflective-AR Methodology

To address perspective ambiguity, the authors propose a novel visualization tool called Reflective-AR displays. This system renders virtual "mirrors" within the user’s field of view, acting as secondary cameras within the virtual scene. These mirrors provide real-time views of the robot from orthogonal angles (e.g., side or top-down views) without requiring the user to physically move around the workspace.

Imaging Geometry

The mirrors are constructed using a "reversed frustum" technique. The system captures the user's current view, flips the image plane, and projects the 3D virtual structures into it. This creates a geometrically accurate, mirror-like experience that allows the user to verify alignment across multiple planes simultaneously, effectively transforming a complex 3D depth estimation task into a series of simpler 2D alignment tasks.

The ViRAAl Algorithm

To ensure the virtual world is accurately registered to the physical environment, the researchers developed the ViRAAl (Virtual-Real Active Alignment) algorithm. This approach treats the human user as the registration sensor. The user performs several interactive alignments, and the system calculates a mathematically robust "mean" transformation.

Because 3D rotations and translations do not exist in a simple linear space, standard arithmetic averaging is insufficient. Instead, the authors utilize Riemannian averaging on the $SE(3)$ manifold (the Special Euclidean group). By performing calculations within Lie algebra, the system ensures that the resulting average transformation is a valid rigid-body pose that minimizes the distance to all individual user trials.

Experimental Evaluation

The methodology was evaluated through a user study involving eight participants. The experimental setup included:

  • Hardware: A 7-DOF KUKA LBR iiwa robotic arm and a first-generation Microsoft HoloLens.
  • Task: A simulated surgical setup involving a mannequin and a trocar (surgical port) placement task.
  • Ground Truth: Measured via 3D landmarks identified through the HoloLens gaze cursor, with a verified measurement error of approximately 3.6 mm.

Results and Key Insights

The study yielded several significant findings regarding the efficacy of multi-view AR guidance:

  1. Significant Error Reduction: Using standard AR overlays resulted in a mean alignment error of 30.2 mm. The introduction of Reflective-AR mirrors reduced this error to 16.5 mm. When combined with the ViRAAl averaging algorithm, the error dropped further to 11.3 mm.
  2. Superiority Over Fixed Monitors: In joint positioning tasks, AR-guided users were vastly more accurate than those using a traditional fixed monitor baseline. The fixed monitor group exhibited a massive 71.4° mean joint error, compared to approximately 24° for the AR-assisted groups.
  3. Joint-Specific Performance: The researchers observed that users are significantly better at aligning "revolving" joints (where the axis is orthogonal to the link) than "twisting" joints (where the axis is parallel to the link), likely due to the visual symmetry inherent in twisting motions.

Discussion and Limitations

While Reflective-AR significantly improves accuracy, the study identified several practical constraints. The first-generation HoloLens's limited field of view (FOV) made it challenging to view multiple mirrors and the physical robot simultaneously without frequent head movement. Additionally, the system is susceptible to SLAM (Simultaneous Localization and Mapping) drift inherent in the headset's spatial mapping.

There is also a trade-off regarding "junk time." The initial registration and alignment process takes roughly 2 to 4 minutes. However, the authors argue that this one-time investment is preferable to the much longer delays caused by mid-surgery collisions or the need to undock and reposition the robot during a procedure.

Conclusion

Reflective-AR and the ViRAAl algorithm provide a robust framework for improving the precision of robotic setup in the operating room. By leveraging multi-perspective visualization and mathematically rigorous averaging, the system overcomes the inherent limitations of human depth perception in AR. Future advancements in HMD hardware, such as wider fields of view and improved eye-tracking, are expected to further enhance the utility and seamlessness of this methodology.

Summary updated

Abstract

Robot-assisted minimally invasive surgery has shown to improve patient outcomes, as well as reduce complications and recovery time for several clinical applications. While increasingly configurable robotic arms can maximize reach and avoid collisions in cluttered environments, positioning them appropriately during surgery is complicated because safety regulations prevent automatic driving. We propose a head-mounted display (HMD) based augmented reality (AR) system designed to guide optimal surgical arm set up. The staff equipped with HMD aligns the robot with its planned virtual counterpart. In this user-centric setting, the main challenge is the perspective ambiguities hindering such collaborative robotic solution. To overcome this challenge, we introduce a novel registration concept for intuitive alignment of AR content to its physical counterpart by providing a multi-view AR experience via reflective-AR displays that simultaneously show the augmentations from multiple viewpoints. Using this system, users can visualize different perspectives while actively adjusting the pose to determine the registration transformation that most closely superimposes the virtual onto the real. The experimental results demonstrate improvement in the interactive alignment of a virtual and real robot when using a reflective-AR display. We also present measurements from configuring a robotic manipulator in a simulated trocar placement surgery using the AR guidance methodology.
BibTeX
@misc{fotouhi2019reflective,
  author = {Javad Fotouhi and Tianyu Song and Arian Mehrfard and Giacomo Taylor and Alejandro Martin-Gomez and Bernhard Fuerst and Mehran Armand and Mathias Unberath and Nassir Navab},
  title = {{Reflective-AR Display: An Interaction Methodology for Virtual-Real Alignment in Medical Robotics}},
  year = {2019},
  eprint = {1907.10138},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.10138},
}
Figure