Human-Robot Interaction in a Shared Augmented Reality Workspace

Shuwen Qiu, Hangxin Liu, Zeyu Zhang, Yixin Zhu, Song-Chun Zhu

Abstract

Traditional Augmented Reality (AR) in robotics often suffers from information asymmetry, where humans perceive a digital layer that remains invisible to the robot. This paper presents a shared AR workspace that bridges this gap, allowing robots to "see" and interact with virtual holograms alongside human partners. By incorporating Level 1 Visual Perspective Taking (VPT1), the robot can estimate the visual cost for a human to perceive specific objects and proactively assist with difficult tasks. Experimental results from a resource collection game demonstrate that this shared perception significantly improves task efficiency and reduces performance variance compared to human-only efforts.

Introduction

In current AR-assisted robotic systems, the human typically acts as the primary agent, issuing commands to a robot that lacks awareness of the human's augmented environment. This one-sided relationship limits the robot's ability to act as a true teammate. To achieve effective human-robot collaboration, both agents must share a common understanding of both the physical and virtual worlds. This research addresses the challenge of information asymmetry by creating a bidirectional interface where robots can perceive, process, and manipulate virtual objects (holograms) just as they do physical ones.

System Architecture and Methodology

The proposed system utilizes a TurtleBot2 mobile robot equipped with a Kinect v2 sensor and Lidar, paired with a Microsoft HoloLens worn by the human user. Real-time data exchange between the HoloLens (Unity3D) and the robot (ROS) is facilitated via ROS#.

Overview of the shared AR workspace system, illustrating the robot platform, human pose detection, and the integration of virtual objects into the shared perception space.

Shared Perception and Manipulation

To enable the robot to process virtual content, 3D hologram meshes are converted into point clouds. This allows the robot’s existing perception and navigation stacks to treat virtual objects as physical entities. Coordinate frames are synchronized across the robot, human, and virtual environment. Shared manipulation is achieved through a proximity-based "attachment" mechanism: when the robot's footprint intersects with a hologram's location, the object is virtually attached to the robot for transport.

Visual Perspective Taking (VPT1)

A core innovation of this work is the implementation of Level 1 Visual Perspective Taking (VPT1). The robot calculates a "visual cost" for the human to see a specific hologram based on their head pose and gaze.

Visual Perspective Taking (VPT1) model categorizing the environment into focusing, transition, and blocked zones based on human gaze and physical/virtual occlusions.

The environment is segmented into three zones:

  1. Focusing: Objects directly within the human's primary field of view.
  2. Transition: Objects requiring a head turn to become visible.
  3. Blocked: Objects occluded by physical structures (e.g., under a table) or other virtual obstacles.

By understanding these perspectives, the robot can prioritize fetching objects that are difficult for the human to locate or reach, moving from a reactive tool to a proactive partner.

Experimental Evaluation

The system was evaluated using a "Resource Collection Game" involving 20 participants. Subjects were tasked with collecting six virtual holograms—some easily visible and others hidden under furniture—and moving them to a target location. Participants were divided into a "Human-only" control group and a "Human+Robot" collaborative group.

Key Results

The collaborative approach yielded significant performance improvements:

  • Efficiency: The Human+Robot group completed tasks in an average of 135 seconds, compared to 202 seconds for the Human-only group ($p = 0.028$).
  • Consistency: The robot's assistance notably reduced the variance in completion times. While human-only performance varied widely depending on how quickly they found occluded objects, the robot's proactive search for high-visual-cost items stabilized performance.
  • Proactive Assistance: Qualitative observations confirmed that the robot successfully identified and retrieved "hidden" items (e.g., an occluded cabbage) while the human focused on more accessible targets.

Conclusion and Future Work

This work demonstrates that shared AR perception and perspective-taking are vital for effective human-robot teaming. By treating holograms as "first-class citizens" in the robot's world model, the system enables proactive collaboration. Future research will explore the use of mobile manipulators for more complex 3D grasping of virtual objects, scaling to multi-agent environments, and integrating haptic feedback to enhance the tangibility of the shared workspace.

Summary updated

BibTeX
@inproceedings{qiu2020human,
  author = {Shuwen Qiu and Hangxin Liu and Zeyu Zhang and Yixin Zhu and Song-Chun Zhu},
  title = {{Human-Robot Interaction in a Shared Augmented Reality Workspace}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=0rFpO9qKPlH},
}
Figure