GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality

Xiyun Hu, Dizhi Ma, Fengming He, Zhengzhe Zhu, Shao-Kang Hsia, Chenfei Zhu, Ziyi Liu, Karthik Ramani

Abstract

Large Language Model (LLM)-based copilots have shown great potential in Extended Reality (XR) applications. However, the user faces challenges when describing the 3D environments to the copilots due to the complexity of conveying spatial-temporal information through text or speech alone. To address this, we introduce GesPrompt, a multimodal XR interface that combines co-speech gestures with speech, allowing end-users to communicate more naturally and accurately with LLM-based copilots in XR environments. By incorporating gestures, GesPrompt extracts spatial-temporal reference from co-speech gestures, reducing the need for precise textual prompts and minimizing cognitive load for end-users. Our contributions include (1) a workflow to integrate gesture and speech input in the XR environment, (2) a prototype VR system that implements the workflow, and (3) a user study demonstrating its effectiveness in improving user communication in VR environments.

Introduction

While Large Language Models (LLMs) have become powerful assistants for coding and writing, using them to manipulate 3D environments in Extended Reality (XR) remains cumbersome. Describing precise 3D coordinates, complex rotations, or specific movement paths using only voice or text is exhausting and often inaccurate—a problem researchers call "pronoun ambiguity." For instance, when a user says "put that over there," an LLM lacks the context to identify the object or the target location.

Currently, users must either use rigid, engineered voice commands or perform tedious "trial-and-error" prompting. GesPrompt addresses this by leveraging natural human communication: using hands to provide the spatial context that words lack.

System Architecture

The GesPrompt workflow treats speech and gesture as a unified input stream. The system consists of three main components:

  1. LLM System (GPT-4o): This module converts the user's speech into text with precise word-level timestamps. It identifies the intended function (e.g., move_object) and flags "ambiguous parameters"—words like "this," "here," or "that way"—that require visual context.
  2. Gesture Processor: When the LLM flags an ambiguous word, the Gesture Processor analyzes skeletal hand data recorded at that specific timestamp. It extracts spatial values based on the gesture type: deictic gestures (pointing) for positions or objects, and iconic gestures for size, orientation, or movement paths.
  3. XR System: This back-end executes the final command in a Unity-based VR environment once all parameters are resolved.

Key Insights and Evaluation

A core innovation of GesPrompt is using speech as a heuristic for gesture processing. The LLM acts as a high-level parser, telling the system when to look at the hands and what to look for (e.g., if the user says "rotate," the system calculates palm rotation). By using word-level timestamps, the system solves the segmentation problem—knowing exactly when a gesture starts and ends—which is a major hurdle in gesture recognition research.

User Study Results

The researchers evaluated GesPrompt with 12 participants across various tasks, including object selection and complex interior design:

  • Efficiency: For complex tasks, GesPrompt was significantly faster (avg. 43.65s) than voice-only (108.58s) or gesture-only (91.11s) baselines.
  • Accuracy: The system achieved high precision, with an average position error of 3.87 cm and a direction error of 2.37°.
  • Usability: Participants rated GesPrompt as more intuitive, scoring 86/100 on the System Usability Scale (SUS), compared to 59/100 for voice-only interaction.

Limitations and Future Work

Current limitations include a lack of real-time feedback; the system waits for the user to finish speaking before processing. Future iterations could provide "ghosting" or real-time visual previews. Additionally, the system can struggle with extremely brief "elliptical" sentences (e.g., just saying "There!") where parameter tokens are missing. Finally, integrating a physics-based validation layer would prevent the LLM from suggesting spatially invalid states, such as placing objects inside walls or floating in mid-air.

Summary updated

BibTeX
@inproceedings{hu2025gesprompt,
  author = {Xiyun Hu and Dizhi Ma and Fengming He and Zhengzhe Zhu and Shao-Kang Hsia and Chenfei Zhu and Ziyi Liu and Karthik Ramani},
  title = {{GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality}},
  booktitle = {Conference on Designing Interactive Systems},
  year = {2025},
  doi = {10.1145/3715336.3735769},
}

Loading related content...

Figure