Language and Gesture in Virtual Reality: Is a Gesture Worth1000 Words?

Padraig Higgins, Cory J. Hayes, Stephanie M. Lukin, Cynthia MatuszekAAAI 2025 · DOI

Abstract

Robots are increasingly incorporating multimodal information and human signals to resolve ambiguity in embodied human-robot interaction. Harnessing signals such as gestures may expedite robot exploration in large, outdoor urban environments for supporting disaster recovery operations, where speech may be unclear due to noise or the challenges of a dynamic and dangerous environment. Despite this potential, capturing human gesture and properly grounding it to crowded, outdoor environments remains a challenge. In this work, we propose a method to model human gesture and ground it to spoken language instructions given to a robot for execution in large spaces. We implement our method in virtual reality to develop a workflow for faster future data collection. We present a series of proposed experiments that compare a language-only baseline to our proposed language supplemented by gesture approach, and discuss how our approach has the potential to reinforce the human’s intent and detect discrepancies in gesture and spoken instructions in these large and crowded environments.

The Challenge of Ambiguity in HRI

In human-robot interaction (HRI), relying solely on natural language is often inefficient. If a human tells a robot to "go to the car" in a crowded parking lot, the instruction is inherently ambiguous. To resolve this using only speech, the human must provide highly specific, wordy descriptions (e.g., "the red sedan third from the left"), which increases cognitive effort and takes valuable time—especially in high-stakes "dirty, dull, or dangerous" environments like search-and-rescue zones.

Comparison of language-only vs. multimodal input in a simulated environment.

While previous research has explored using gestures (like pointing) to ground language, most studies are limited to "tabletop" settings—small-scale interactions where the robot and human are inches apart. Scaling this to large, outdoor-style environments is difficult because of the distance involved, the potential for noisy speech, and the difficulty of keeping both the human and the target object in a single camera frame.

A Multimodal Gesture and Language Pipeline

The authors propose a modular pipeline integrated with the Robot Operating System (ROS) to address these challenges. The system is designed to handle the complexities of large-scale environments through several key components:

  • Speech & Language Processing: Spoken commands are transcribed via OpenAI’s Whisper and converted into Abstract Meaning Representation (AMR) graphs. This normalizes different phrasing into a standard format the robot can process.
  • World Modeling: The robot utilizes YOLOv8 for object detection and CLIP embeddings to associate visual data with semantic labels.
  • 3D Gesture Recognition: The system uses 3D pose estimation to track the human’s shoulder, elbow, and wrist, creating a "pointing ray" in 3D space.
  • Bayesian Grounding & Interest Metric: The system calculates the probability that a specific object is the target based on its angular distance from the pointing ray. To filter out accidental movements, an "Interest" metric accumulates probability over the duration of the spoken utterance. If an object maintains high "interest" for more than 0.3 seconds, it is considered the intended target.
  • VR-Based Workflow: The entire system is implemented in Unity using Meta Quest 2 headsets. This allows for the simulation of complex outdoor environments and rapid collection of human-gesture data safely before moving to physical robots.

Key Innovations: Culling and Conflict Detection

The pipeline introduces several novel concepts to improve interaction efficiency:

  1. Gestures as a "Cull" Mechanism: Instead of the gesture being the primary command, it acts as a filter. If the human points, the robot "culls" (ignores) objects far from the pointing ray, drastically narrowing the search space for the language model.
  2. Conflict Detection: The pipeline is designed to detect discrepancies. If a user says "Go to the car on the left" but points to the right, the Dialogue Manager recognizes the mismatch and triggers a clarification request.
  3. The "Interest" Accumulator: By multiplying probability by time, the system avoids "jitter" issues common in pose tracking. It requires the human to dwell on a target, mimicking natural human intent.

Validation and Future Work

As a "work in progress," the paper focuses on system architecture and proof-of-concept validation. Mathematical analysis demonstrates that the Bayesian approach effectively differentiates between objects based on angular distance. For instance, an object at a 16.7° angle from the pointing ray receives a high probability (0.355), while an object at 31.9° is assigned a probability of 0.0, effectively culling it.

Future work aims to bridge the "sim-to-real" gap, transitioning from high-fidelity VR tracking to real-world robot onboard RGB-D cameras. Additionally, the researchers plan to address environmental noise in disaster zones and adapt the world model to handle dynamic, moving environments.

Summary updated

Abstract

Robots are increasingly incorporating multimodal information and human signals to resolve ambiguity in embodied human-robot interaction. Harnessing signals such as gestures may expedite robot exploration in large, outdoor urban environments for supporting disaster recovery operations, where speech may be unclear due to noise or the challenges of a dynamic and dangerous environment. Despite this potential, capturing human gesture and properly grounding it to crowded, outdoor environments remains a challenge. In this work, we propose a method to model human gesture and ground it to spoken language instructions given to a robot for execution in large spaces. We implement our method in virtual reality to develop a workflow for faster future data collection. We present a series of proposed experiments that compare a language-only baseline to our proposed language supplemented by gesture approach, and discuss how our approach has the potential to reinforce the human’s intent and detect discrepancies in gesture and spoken instructions in these large and crowded environments.
BibTeX
@inproceedings{higgins2025language,
  author = {Padraig Higgins and Cory J. Hayes and Stephanie M. Lukin and Cynthia Matuszek},
  title = {{Language and Gesture in Virtual Reality: Is a Gesture Worth1000 Words?}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaaiss.v7i1.36947},
}
Figure