Abstract
Effective human-computer interaction requires AI systems to master multimodal communication, specifically the simultaneous use of speech and gesture. This paper introduces the Embodied Multimodal Referring Expressions (EMRE) dataset, a resource containing over 2,000 human evaluations of synthetic referring behaviors. Our analysis demonstrates that "ensemble" communication—combining linguistic descriptors with physical gestures—is significantly more natural and effective than using either modality in isolation.
Introduction
When humans communicate in physical spaces, we rarely rely on words alone. A request like "Give me that green block" is typically accompanied by a pointing gesture to resolve ambiguity. While modern AI systems have improved at interpreting such multimodal inputs, they struggle to generate them naturally. Most virtual assistants are either overly verbose or use ambiguous gestures, failing to strike the human-like balance between "this" and "that."
Methodology
To study these dynamics, we utilized VoxSim, a 3D visual event simulator, to create a controlled environment for a virtual avatar.
- Scene Configuration: We generated 50 distinct configurations featuring six colored blocks (red, green, purple) and three landmark objects (a plate, knife, and cup).
- Referencing Strategies: The avatar identified target blocks using three distinct modes: Gestural (pointing), Linguistic (spatial phrases), and Ensemble (combining both).
- Experimental Variables: We varied the number of relational descriptors (0–3) and tested two distance logics: Absolute (near/far zones) and Relative (comparative distance between similar objects).
The EMRE Dataset
The resulting Embodied Multimodal Referring Expressions (EMRE) dataset consists of 1,500 video references. Unlike static image datasets, EMRE captures the temporal alignment of speech and movement in 3D space. Each entry includes full spatial coordinates, Euclidean distances, and relational descriptors based on Region Connection Calculus. Crucially, the dataset includes 2,228 high-quality human naturalness assessments collected via Amazon Mechanical Turk.
Key Insights
Our analysis of the human evaluation data revealed several critical patterns in how multimodal communication is perceived:
- The Ensemble Advantage: Participants overwhelmingly preferred the combination of speech and gesture. Purely gestural references were consistently rated as the least natural, highlighting that pointing alone is often too ambiguous in complex scenes.
- Optimal Conciseness: We observed a "sweet spot" for linguistic detail. While three descriptors provided maximum clarity, a single descriptor paired with a gesture was often rated higher than two descriptors. This suggests that gestures allow for—and humans prefer—greater linguistic brevity.
- Spatial Demarcation: "Absolute" distance distinctions (splitting the workspace into "near" and "far") were perceived as more natural than "relative" comparisons. This indicates that humans may use static mental boundaries when choosing between demonstratives like "this" and "that."
Results
The human evaluation scores confirm the superiority of the ensemble approach. As shown in the distribution of score offsets, the ensemble method consistently achieved the highest probability of being rated above the median naturalness score.
In contrast, gesture-only references frequently received the lowest scores, reinforcing the insufficiency of non-verbal cues in isolation.
Conclusion and Future Work
The EMRE dataset provides a statistically significant baseline for generating synthetic multimodal behavior. While currently limited in scale and environmental complexity, it lays the groundwork for AI that can interact within shared physical or virtual spaces. Future work will focus on expanding the corpus to include "busy hands" scenarios—where the avatar cannot point—and developing real-time machine learning models that can dynamically generate these referring expressions based on live scene analysis.
Summary updated