Multimodal Grounding of Deictic Expressions

18 concepts14 papers1 proposal

Imagine a human collaborator pointing toward a cluttered table and asking, "Can you hand me that?" To a text-only language model, this sentence is an unresolvable riddle. The word "that" contains zero inherent semantic information about the object's shape, color, or utility. Yet, humans resolve these ambiguities instantly. We do this by seamlessly fusing the temporal stream of spoken language with spatial and physical context—a process known as deictic reference resolution.

Deictic expressions (words like this, that, here, and it) are the bedrock of natural, situated communication. However, building computational systems capable of grounding these expressions requires moving beyond independent text or image processing. Instead, it demands a tight, real-time coupling of linguistic tokens with physical gestures, gaze, and spatial environments.

The Symbiosis of Speech and Gesture

Why can't we just resolve language first and look at gestures later? The answer lies in the deeply intertwined nature of human communication modalities. As Dagmar Schmauks observed in Natural and Simulated Pointing, physical pointing is inherently imprecise and context-dependent, while verbal descriptions are frequently underspecified.

The two modalities operate symmetrically to patch each other's vulnerabilities:

  1. Gestures simplify language: Instead of constructing a dense, highly descriptive noun phrase ("the small, slightly tarnished silver wrench on the bottom left shelf"), a speaker can simply say "this" while pointing.
  2. Language disambiguate gestures: If a user points vaguely toward a cluster of objects, accompanying verbal cues ("the red one") narrow down the intended target.

Without integrating both streams, interactive systems are blind to user intent. Fusing them allows the system to leverage discourse memory and spatial context to resolve inputs that would be completely uninterpretable in isolation.

Early Foundations: Mapping and Planning

The challenge of multimodal integration was recognized long before modern deep learning. In the late 1970s, David C. Brown et al. pioneered NLG - Natural Language Graphics, a system designed to synchronize linguistic tokens with physical screen inputs, mapping coordinates directly to words like "here."

As the field progressed, researchers realized that spatial coordinates alone were insufficient; systems needed to track the user's cognitive focus. Oliviero Stock demonstrated in Natural Language and Exploration of an Information Space: The ALFresco Interactive System that resolving ambiguous touchscreen gestures requires a dynamic model of the ongoing discourse context.

This problem is equally vital when an AI needs to speak to a human. To generate natural explanations, an agent must reverse the resolution process. As Mark T. Maybury argued in Planning Multimedia Explanations Using Communicative Acts, an automated system cannot effectively plan when to use deictic terms like "this" or "there" unless it maintains strict internal awareness of the user's current visual focus and the spatial state of the display.

From Rules to Statistical Salience

As the field transitioned away from rigid, rule-based systems, researchers began treating gestures as statistical features within broader linguistic models. Jacob Eisenstein et al. demonstrated in Gesture Improves Coreference Resolution that computer-vision-derived gesture features provide statistically significant cues that directly improve the accuracy of coreference resolution algorithms.

The authors extended this paradigm in Turning Lectures into Comic Books Using Linguistically Salient Gestures by treating gesture salience as a hidden variable. By identifying which physical actions mathematically clarified the accompanying spoken text, their system could automatically pair key video frames with transcripts.

This line of work proved that physical context is not an optional "add-on" to language; rather, human speakers strategically optimize their choice of referring expressions based on shared physical environments and discourse salience, a principle formalized by Naho Orita et al. in Why discourse affects speakers' choice of referring expressions.

+-----------------------------------------------------------------------+
|                          SITUATED DIALOGUE                            |
|                                                                       |
|  [ Linguistic Stream ]  =======>  "Is THIS it?"                       |
|                                       ||                              |
|                                       || (Real-Time Fusion)           |
|                                       \/                              |
|  [ Physical Stream ]    =======>  [Gaze/Pointing Vector] -> (Object)  |
+-----------------------------------------------------------------------+

Why Modern Vision-Language Models Struggle

Given the success of modern, massive Vision-Language Models (VLMs), one might assume that deictic reference resolution is a solved problem. However, recent evaluations reveal a significant performance gap between static image comprehension and real-time situated interaction.

In Can Vision-Language Models Answer Face to Face Questions in the Real-World?, Reza Pourreza et al. exposed a critical flaw: state-of-the-art VLMs struggle with real-time, face-to-face questions because they typically process visual and linguistic inputs as decoupled, parallel streams rather than integrated, co-occurring signals. They lack the fine-grained temporal reasoning required to map a specific syllable to a fleeting physical gesture.

Furthermore, standard AI benchmarks rarely mirror the messy reality of human collaboration. As Sean Andrist et al. argued in "Is This It?": Towards Ecologically Valid Benchmarks for Situated Collaboration, real-world physical collaboration is saturated with highly ambiguous deictic expressions. In active environments, pronouns like "it" or "this" appear far more frequently than in static observation datasets, demanding an exceptionally tight coupling between real-time visual perception and language processing.

Establishing Common Ground in Immersive Spaces

To achieve true collaboration in complex environments—such as virtual reality, training simulators, or physical workspaces—an AI must establish "common ground." This is the shared understanding of the workspace state between the human and the machine.

Without this cross-modal grounding, conversational agents cannot verify if they are actually referring to the same entities as the user. Karl Schultz et al. highlighted this limitation in Interactive Spoken Simulation Control and Conversational Tutoring, noting that speech-only tutoring systems fail when tasks require a shared spatial focus.

This challenge becomes acute in immersive environments where the user's entire field of view is a variable:

By treating language not as an isolated text string, but as a single thread woven into a broader tapestry of physical, spatial, and temporal signals, modern AI systems can begin to understand the world the way humans do: in context.

Key Foundational Work

Natural and Simulated Pointing (1987) — contributed 3 concept elements to this direction.
NLG - Natural Language Graphics (1977) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_deictic_reference_resolution_multimodal__3afc25b5,
  title = {deictic reference resolution (multimodal integration of pointing and speech)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deictic-reference-resolution-multimodal-integration-of-pointing-and-speech/rdr_3afc25b531844011b006cb2573f0c76b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_deictic_reference_resolution_multimodal__3afc25b5_v1,
  title = {deictic reference resolution (multimodal integration of pointing and speech)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/deictic-reference-resolution-multimodal-integration-of-pointing-and-speech/rdr_3afc25b531844011b006cb2573f0c76b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake