Abstract
Human-robot collaboration is an essential research topic in artificial intelligence (AI), enabling researchers to devise cognitive AI systems and affords an intuitive means for users to interact with the robot. Of note, communication plays a central role. To date, prior studies in embodied agent navigation have only demonstrated that human languages facilitate communication by instructions in natural languages. Nevertheless, a plethora of other forms of communication is left unexplored. In fact, human communication originated in gestures and oftentimes is delivered through multimodal cues, e.g. "go there" with a pointing gesture. To bridge the gap and fill in the missing dimension of communication in embodied agent navigation, we propose investigating the effects of using gestures as the communicative interface instead of verbal cues. Specifically, we develop a VR-based 3D simulation environment, named Ges-THOR, based on AI2-THOR platform. In this virtual environment, a human player is placed in the same virtual scene and shepherds the artificial agent using only gestures. The agent is tasked to solve the navigation problem guided by natural gestures with unknown semantics; we do not use any predefined gestures due to the diversity and versatile nature of human gestures. We argue that learning the semantics of natural gestures is mutually beneficial to learning the navigation task--learn to communicate and communicate to learn. In a series of experiments, we demonstrate that human gesture cues, even without predefined semantics, improve the object-goal navigation for an embodied agent, outperforming various state-of-the-art methods.
Introduction
Most research in Vision-Language Navigation (VLN) assumes that humans interact with robots via natural language instructions. While effective, language can be verbose, ambiguous, and sensitive to environmental noise. In contrast, human communication is deeply rooted in nonverbal cues. Gestures are immediate, intuitive, and context-dependent, yet teaching an artificial agent to understand them is challenging due to their diversity and dependence on physical context.
This research addresses the lack of robust frameworks for gesture-based embodied navigation. By introducing Ges-THOR, a simulation framework where agents learn to navigate by interpreting natural human gestures, we explore a "learn to communicate and communicate to learn" approach. Here, gesture semantics are learned implicitly alongside the navigation task, rather than being hard-coded.
Methods
The researchers developed Ges-THOR, an interactive learning environment built on the AI2-THOR platform. To collect realistic data, human volunteers were immersed in a virtual scene using an Oculus Rift for first-person perspective, a Kinect for body pose tracking, and a Leap Motion controller for fine-grained hand gestures.
The navigation agent, a capsule-shaped robot, is trained using Proximal Policy Optimization (PPO). Its policy network fuses three distinct inputs:
- Visual: RGB-D images processed via a ResNet-18 backbone.
- Categorical: The target object type (e.g., "bread").
- Gestural: A sequence of 100 time-steps representing 95 skeletal and hand features.
A core innovation is the use of implicit semantics. The agent is not provided with pre-defined meanings for gestures like pointing. Instead, it must infer intent based on rewards received for reaching target objects, effectively grounding the communication in the physical environment. The study utilized a dataset of over 230,000 unique training episodes across 120 indoor scenes.
Results and Discussion
The agents were evaluated on Success Rate (SR) and Success weighted by Path Length (SPL). The results demonstrated massive performance gains over vision-only baselines. In Kitchen environments, a vision-only agent achieved an 11.5% SR. Incorporating referencing gestures (pointing) increased this to 19.2%.
Most notably, intervention gestures (rejective motions used to warn the agent if it moves away from the goal) caused the SR to skyrocket to 40.3%. This suggests that corrective feedback is significantly more powerful for navigation guidance than simple directional cues. The agent also showed strong generalization capabilities, successfully interpreting gestures in unseen test rooms, indicating it learned the underlying relationship between human motion and spatial goals.
Conclusion
This work demonstrates that natural gestures, even without predefined semantics, significantly enhance object-goal navigation for embodied agents. Future research will focus on bridging the sim-to-real gap and developing bi-directional communication, allowing agents to gesture back to humans for clarification or to signal intent. Expanding the task scope from navigation to complex manipulation remains a key objective for future iterations of the Ges-THOR framework.
Summary updated