A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot

Patrick Stotko, Stefan Krumpen, Max Schwarz, Christian Lenz, Sven Behnke, Reinhard Klein, Michael WeinmannIROS 2019 · DOI · OpenReview

Abstract

Applications like disaster management and industrial inspection often require experts to enter contaminated places. To circumvent the need for physical presence, it is desirable to generate a fully immersive individual live teleoperation experience. However, standard video-based approaches suffer from a limited degree of immersion and situation awareness due to the restriction to the camera view, which impacts the navigation. In this paper, we present a novel VR-based practical system for immersive robot teleoperation and scene exploration. While being operated through the scene, a robot captures RGB-D data that is streamed to a SLAM-based live multiclient telepresence system. Here, a global 3D model of the already captured scene parts is reconstructed and streamed to the individual remote user clients where the rendering for head-mounted display devices (HMDs) is performed. We introduce a novel lightweight robot client component which transmits robot-specific data and enables a quick integration into existing robotic systems. This way, in contrast to first-person exploration systems, the operators can explore and navigate in the remote site completely independent of the current position and view of the capturing robot. We provide a proof-of-concept implementation and demonstrate the capabilities as well as the performance of our system regarding interactive object measurements and bandwidth-efficient data streaming and visualization.

Introduction

Teleoperating a robot in a disaster zone or a contaminated industrial site is notoriously difficult. Most operators rely on a "straw-hole" view—a single video feed that makes judging distances and maintaining situational awareness a constant struggle. This creates two major hurdles: limited immersion, where operators lose the sense of scale and depth, and loss of context, where objects "disappear" from view once the robot turns away.

To address these challenges, we developed a system that swaps this narrow window for a full-blown virtual reality (VR) experience, allowing operators to step "inside" a persistent, real-time 3D reconstruction of the remote environment.

System Architecture and Methods

The system is built upon the SLAMCast framework, creating a seamless pipeline from data capture to immersive visualization.

The system overview illustrates the pipeline from robot capture to VR teleoperation.

The Mobile Platform

Our prototype, "Mario," is a mobile platform equipped with a Kinect v2 sensor mounted on a 6-DoF robotic arm. As the robot moves, the arm automatically scans the environment in a Z-shaped pattern to maximize coverage. We introduced a lightweight "Robot Client" that transmits the robot's joint positions and wheel orientations, allowing the VR system to render an accurate virtual twin of the robot within the 3D scene.

Real-time Reconstruction and Streaming

The system uses dense volumetric SLAM based on voxel block hashing. Instead of simple point clouds, it calculates Truncated Signed Distance Fields (TSDFs) to create a solid 3D surface. To maintain low bandwidth, the server converts the 3D model into Marching Cubes (MC) indices, which are compressed and streamed to the VR client. This method requires a mean bandwidth of only 14 MBit/s, making it viable for operation over standard internet connections.

Immersive Teleoperation in VR

The operator interacts with the environment using an HTC Vive HMD. A key innovation of our system is the decoupling of the operator's viewpoint from the robot's physical camera.

An operator uses the VR system to navigate the reconstructed environment independently of the robot's current view.

By allowing the operator to "teleport" to previously scanned areas or walk around the virtual robot to check its clearance from a third-person perspective, they gain a much better understanding of the surroundings than a first-person view could provide. Furthermore, because the 3D model is stored on the client side, the operator remains immersed in the last known state of the world even if the network connection is temporarily lost.

Results and Key Insights

Our evaluation, including a user study with 20 participants, revealed several key benefits:

  • Situational Awareness: The VR mode scored significantly higher than traditional video-based control for localization and maneuvering around obstacles.
  • Navigation Precision: While users were slightly faster in video mode, they had significantly fewer collisions when using the VR system.
  • Measurement Accuracy: The solid surface reconstruction allows for interactive tasks. Operators could measure objects in the virtual scene (like door widths) with an error of less than 1 cm, which is sufficient for high-stakes rescue management.

Limitations and Future Work

While successful, the system faces challenges such as motion sickness during automatic following, which we currently mitigate through teleportation. Future work will focus on projecting live high-resolution camera textures onto the 3D mesh to improve visual fidelity and further reducing the latency of the reconstruction stream.

Summary updated

Abstract

Applications like disaster management and industrial inspection often require experts to enter contaminated places. To circumvent the need for physical presence, it is desirable to generate a fully immersive individual live teleoperation experience. However, standard video-based approaches suffer from a limited degree of immersion and situation awareness due to the restriction to the camera view, which impacts the navigation. In this paper, we present a novel VR-based practical system for immersive robot teleoperation and scene exploration. While being operated through the scene, a robot captures RGB-D data that is streamed to a SLAM-based live multi-client telepresence system. Here, a global 3D model of the already captured scene parts is reconstructed and streamed to the individual remote user clients where the rendering for e.g. head-mounted display devices (HMDs) is performed. We introduce a novel lightweight robot client component which transmits robot-specific data and enables a quick integration into existing robotic systems. This way, in contrast to first-person exploration systems, the operators can explore and navigate in the remote site completely independent of the current position and view of the capturing robot, complementing traditional input devices for teleoperation. We provide a proof-of-concept implementation and demonstrate the capabilities as well as the performance of our system regarding interactive object measurements and bandwidth-efficient data streaming and visualization. Furthermore, we show its benefits over purely video-based teleoperation in a user study revealing a higher degree of situation awareness and a more precise navigation in challenging environments.
BibTeX
@inproceedings{stotko2019vr,
  author = {Patrick Stotko and Stefan Krumpen and Max Schwarz and Christian Lenz and Sven Behnke and Reinhard Klein and Michael Weinmann},
  title = {{A VR System for Immersive Teleoperation and Live Exploration with a Mobile Robot}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  doi = {10.1109/iros40897.2019.8968598},
}
Figure