Natural language is inherently underspecified. When a human tells a robot to "pick up that gear" or "move it to the left," the verbal instruction lacks the precision required for robotic execution in a physical environment. Without additional context, a robot cannot distinguish between multiple identical objects or define the spatial boundaries of "there." To bridge this gap, research has shifted toward multimodal grounding—the integration of non-verbal cues like pointing, gaze, and sketching to resolve linguistic ambiguity.
The Geometry of Deictic Reference
The most common non-verbal cues are deictic references, such as pointing gestures, which ground abstract words in 3D space. Early systems, such as the one described by Christian Scheering et al. in Dialoggesteuerte Handlungskontrolle eines bimanualen Konstruktionsroboters, recognized that robots struggle with spatial ambiguity when relying solely on verbal instructions.
The intuition behind modern solutions is to treat these cues as geometric constraints. Joshua B. Tenenbaum et al. demonstrate this in Understanding Embodied Reference with Touch-Line Transformer by proposing a "virtual touch line" (VTL). Rather than modeling a gesture as a simple vector from the wrist, the VTL connects the speaker's eye to their fingertip, mimicking how humans actually aim their gaze and hands. This geometric prior allows models to narrow down the "search space" for an object far more effectively than vision-language models that treat the image as a flat 2D plane.
Gaze as a Robustness Layer
While pointing provides a directional vector, gaze serves as a powerful "tie-breaker" for resolving references among similar candidates. In environments where multiple identical objects exist—such as a tray of bolts in a factory—Chiara Falagario et al. show in Multi-Modal Robotic Architecture for Object Referring Tasks Aimed At Designing New Rehabilitation Strategies (short paper) that gaze estimation can maintain high identification accuracy even when verbal descriptions are minimal.
This integration works because gaze reveals the user's internal focus of attention. Jens V. Rüppel et al. leverage this in Gaze-supported Large Language Model Framework for Bi-directional Human-Robot Interaction by using Large Language Models (LLMs) to reason about the relationship between a user's gaze and the objects in a 3D scene. By mapping gaze to specific objects, the system can resolve deictic pronouns like "this" or "that" which are otherwise impossible to ground.
Perspective and Complementary Information
Reference resolution is not just about coordinates; it is about perspective. A command like "the cup to my left" requires the robot to mentally rotate its frame of reference to match the speaker's. Cheng Shi 0001 et al. address this in Spatial and Visual Perspective-Taking via View Rotation and Relation Reasoning for Embodied Reference Understanding by developing a model that performs "perspective-taking," combining 3D depth estimation with multimodal reasoning to understand the speaker's spatial orientation.
Furthermore, gestures often provide non-redundant, representational information that speech lacks. For instance, Esam Ghaleb et al. argue in I see what you mean: Co-Speech Gestures for Reference Resolution in Multimodal Dialogue that representational gestures—like drawing a shape in the air—provide vital geometric details that help machines "see" what a user means when their vocabulary is insufficient.
Enhancing Navigation and Task Execution
The integration of these cues significantly improves the efficiency of embodied agents. In navigation tasks, Qi Wu et al. found in Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene that agents using human gestures reached targets via more direct paths than those using vision alone. The gestures act as immediate, context-dependent guidance that resolves environmental ambiguity.
For complex, multi-step tasks, the authors of NatSGLD: A Dataset with Speech, Gesture, Logic, and Demonstration for Robot Learning in Natural Human-Robot Interaction demonstrate that fusing verbal commands with tacit spatial context (gestures and gaze) allows robots to ground abstract verbs—like "clean" or "cook"—into specific physical locations and objects. This multimodal fusion, as also explored by Md Mofijul Islam et al. in EQA-MX: Embodied Question Answering using Multimodal Expression, ensures that robots do not just hear the instruction, but understand the specific physical intent behind it.
Key Foundational Work
Dialoggesteuerte Handlungskontrolle eines bimanualen Konstruktionsroboters (1998) — contributed 1 concept elements to this direction.
An Integrated Robotic System for Spatial Understanding and Situated Interaction in Indoor Environments (2007) — contributed 1 concept elements to this direction.