XR Prototyping of Mixed Reality Visualizations: Compensating Interaction Latency for a Medical Imaging Robot

Jan Hendrik Plümer, Kevin Yu, Ulrich Eck, Denis Kalkofen, Philipp Steininger, Nassir Navab, Markus Tatzgern

Introduction

Developing medical robotics is a complex, capital-intensive process governed by stringent safety regulations. For large-scale hardware like the Loop-X—a 500kg mobile imaging robot—prototyping new user interfaces and interaction paradigms presents significant logistical hurdles. Researchers often lack consistent access to clinical environments or the physical hardware itself. This study investigates the efficacy of Extended Reality Prototyping (XRP) as a high-fidelity alternative, specifically focusing on whether virtual twins can accurately predict the performance of visualization strategies designed to mitigate mechanical latency.

The Challenge: Interaction Latency

A primary hurdle in operating heavy medical robots is interaction latency. Due to their immense weight and the safety protocols required for movement, there is a distinct delay between a user’s joystick input and the physical response of the robot’s wheels. This lag can lead to overcorrection, user frustration, and decreased precision during critical medical procedures. To address this, researchers proposed using Mixed Reality (MR) visualizations to "mask" or compensate for this latency, using an XRP framework to test these designs before full-scale implementation.

Methodology

The researchers conducted a within-subject study involving 24 participants using a Varjo XR-3 headset. The study compared two primary environments:

  1. REAL: Interaction with the actual Loop-X robot in a physical space.
  2. XRP: Interaction with a high-fidelity digital twin in a simulated environment.

Within both environments, three distinct visualization strategies were evaluated for their ability to manage mechanical lag:

  • Direct Control (DIR): The baseline condition where users move the joystick and wait for the physical/simulated robot to respond.
  • Steering Visualization (STE): An augmented reality "progress bar" for the wheels. Virtual lines are projected onto the floor to show the current rotation of the hidden wheels, providing immediate visual feedback that the system has received the user's command.
  • Surrogate Control (SUR): Users control a "ghost" version of the robot that responds instantaneously to input. The physical or simulated robot then autonomously follows the path established by the ghost.

Performance was measured through Task Completion Time (TCT), while subjective experience was captured via the NASA-TLX (workload), System Usability Scale (SUS), and qualitative interviews.

Key Insights: Relative vs. Absolute Validity

The study’s most significant conceptual finding is the distinction between absolute and relative validity in XR simulations.

  • Absolute Validity: The simulation did not perfectly mirror real-world performance. Participants were generally faster in the XRP environment than in the REAL environment.
  • Relative Validity: Despite the difference in raw times, the ranking of the three visualization methods was identical across both environments. The performance gains observed in the simulation translated directly to the real world.

This confirms that while XRP might not predict exact real-world completion times, it is an exceptionally reliable tool for determining which design intervention is superior.

Results and Performance Gains

The Surrogate (SUR) method emerged as the most effective strategy. By decoupling user input from the slow physical movement of the robot, the system felt more responsive. In the REAL condition, the Surrogate mode reduced the mean task completion time from approximately 135 seconds (DIR) to 86 seconds.

User preference mirrored these objective results: 54% of participants ranked the Surrogate method as their top choice, followed by 38% for Steering Visualization. Only 8% preferred the baseline Direct Control. Furthermore, the SUS scores for the simulation were consistent with the real-world setup, indicating that the XRP framework successfully captured the "feel" and usability of the interface.

The "Fear Factor" and Simulation Gaps

Qualitative data revealed a psychological gap between the two environments. Participants reported higher stress and mental demand when operating the actual 500kg robot. In the simulation, users were "bolder," whereas in the real world, the fear of colliding with expensive equipment or surgical tables led to more cautious, slower movements.

Additionally, some users expressed a lack of "Sense of Agency" or trust when using the Surrogate method with the real robot. Because the robot followed the ghost autonomously, some participants felt insecure, fearing the robot might deviate or collide with objects while they weren't in "direct" control.

Conclusion

This research demonstrates that Extended Reality Prototyping is a powerful tool for the medical robotics industry. By providing high relative validity, XRP allows designers to iterate on complex visualization strategies—like the highly successful Surrogate Control—without the need for constant access to physical hardware. Future work will focus on integrating human behavior models, such as trust and agency, into the XRP framework to further bridge the gap between virtual prototyping and clinical reality.

Summary updated

BibTeX
@inproceedings{plumer2024xr,
  author = {Jan Hendrik Plümer and Kevin Yu and Ulrich Eck and Denis Kalkofen and Philipp Steininger and Nassir Navab and Markus Tatzgern},
  title = {{XR Prototyping of Mixed Reality Visualizations: Compensating Interaction Latency for a Medical Imaging Robot}},
  booktitle = {ISMAR},
  year = {2024},
  doi = {10.1109/ismar62088.2024.00014},
}
Figure