Abstract
Traditional Augmented Reality (AR) in robotics often suffers from information asymmetry, where humans perceive a digital layer that remains invisible to the robot. This paper presents a shared AR workspace that bridges this gap, allowing robots to "see" and interact with virtual holograms alongside human partners. By incorporating Level 1 Visual Perspective Taking (VPT1), the robot can estimate the visual cost for a human to perceive specific objects and proactively assist with difficult tasks. Experimental results from a resource collection game demonstrate that this shared perception significantly improves task efficiency and reduces performance variance compared to human-only efforts.
Introduction
In current AR-assisted robotic systems, the human typically acts as the primary agent, issuing commands to a robot that lacks awareness of the human's augmented environment. This one-sided relationship limits the robot's ability to act as a true teammate. To achieve effective human-robot collaboration, both agents must share a common understanding of both the physical and virtual worlds. This research addresses the challenge of information asymmetry by creating a bidirectional interface where robots can perceive, process, and manipulate virtual objects (holograms) just as they do physical ones.
System Architecture and Methodology
The proposed system utilizes a TurtleBot2 mobile robot equipped with a Kinect v2 sensor and Lidar, paired with a Microsoft HoloLens worn by the human user. Real-time data exchange between the HoloLens (Unity3D) and the robot (ROS) is facilitated via ROS#.
Shared Perception and Manipulation
To enable the robot to process virtual content, 3D hologram meshes are converted into point clouds. This allows the robot’s existing perception and navigation stacks to treat virtual objects as physical entities. Coordinate frames are synchronized across the robot, human, and virtual environment. Shared manipulation is achieved through a proximity-based "attachment" mechanism: when the robot's footprint intersects with a hologram's location, the object is virtually attached to the robot for transport.
Visual Perspective Taking (VPT1)
A core innovation of this work is the implementation of Level 1 Visual Perspective Taking (VPT1). The robot calculates a "visual cost" for the human to see a specific hologram based on their head pose and gaze.
The environment is segmented into three zones:
- Focusing: Objects directly within the human's primary field of view.
- Transition: Objects requiring a head turn to become visible.
- Blocked: Objects occluded by physical structures (e.g., under a table) or other virtual obstacles.
By understanding these perspectives, the robot can prioritize fetching objects that are difficult for the human to locate or reach, moving from a reactive tool to a proactive partner.
Experimental Evaluation
The system was evaluated using a "Resource Collection Game" involving 20 participants. Subjects were tasked with collecting six virtual holograms—some easily visible and others hidden under furniture—and moving them to a target location. Participants were divided into a "Human-only" control group and a "Human+Robot" collaborative group.
Key Results
The collaborative approach yielded significant performance improvements:
- Efficiency: The Human+Robot group completed tasks in an average of 135 seconds, compared to 202 seconds for the Human-only group ($p = 0.028$).
- Consistency: The robot's assistance notably reduced the variance in completion times. While human-only performance varied widely depending on how quickly they found occluded objects, the robot's proactive search for high-visual-cost items stabilized performance.
- Proactive Assistance: Qualitative observations confirmed that the robot successfully identified and retrieved "hidden" items (e.g., an occluded cabbage) while the human focused on more accessible targets.
Conclusion and Future Work
This work demonstrates that shared AR perception and perspective-taking are vital for effective human-robot teaming. By treating holograms as "first-class citizens" in the robot's world model, the system enables proactive collaboration. Future research will explore the use of mobile manipulators for more complex 3D grasping of virtual objects, scaling to multi-agent environments, and integrating haptic feedback to enhance the tangibility of the shared workspace.
Summary updated