Multimodal Semantic Simulations of Linguistically Underspecified Motion Events

Nikhil Krishnaswamy, James Pustejovsky

Introduction

Human language is inherently underspecified. When a person is told to "put the knife on the table," they intuitively understand the orientation, the target surface, and the physical constraints involved. For a computer, however, such a command lacks the low-level geometric data—translations, rotations, and physics parameters—necessary to render a plausible 3D simulation. Researchers at Brandeis University have addressed this gap with Voxicon, a system designed to transform high-level natural language into dynamic, physically grounded 3D animations.

The Voxicon Architecture

The Voxicon system bridges linguistic theory and game engine technology by treating verbs as executable programs and objects as data-rich entities. The pipeline utilizes external NLP tools for initial parsing, which then feeds into a specialized architecture within the Unity game engine.

The Voxicon system architecture, illustrating the flow from parsing to the Unity-based Simulator, Program Composer, and Object Composer.

As shown in the architecture diagram, the Program Composer and Object Composer work in tandem within Unity to resolve linguistic ambiguity. The system relies on VoxML (Visual Object Markup Language), which defines objects as "voxemes." Unlike standard 3D models, voxemes include:

  • Habitats: Intrinsic physical constraints (e.g., a table's surface is its "top").
  • Affordances: Functional data describing what can be done with an object (e.g., a cup is for "containing").

Operationalizing Motion and Spatial Logic

To simulate motion, Voxicon treats verbs as code using Dynamic Interval Temporal Logic (DITL). A complex action like "put" is decomposed into a sequence of primitive operations: grasp → move → ungrasp.

Spatial relationships are managed via Region Connection Calculus (RCC8). This allows the system to distinguish between subtle prepositional meanings. For example, the system determines the difference between "in the cup" and "on the cup" by querying the cup's geometry and its specific "containment" affordance. Because the system understands that a cup is a container, it can automatically align a knife's longest dimension to fit inside the cup's concavity when commanded to "put the knife in the cup."

Results and Handling Underspecification

The system demonstrates a high degree of qualitative success in generating simulations that respect both the speaker's intent and physical reality. By querying an object's "habitat," the system can adapt the meaning of prepositions like "on" based on the target object's geometry.

Simulated motion events demonstrating how Voxicon handles underspecified commands for a knife interacting with a table, a container, and a plate.

As illustrated above, the system successfully differentiates between placing a knife on a flat surface (table or plate) versus inside a container. It resolves the "top" of the table and the orientation of the knife without explicit instructions, filling in the blanks of the underspecified text.

Conclusion and Future Directions

Voxicon represents a significant step forward in text-to-visual systems by moving beyond static scenes to dynamic, physics-based motion. While currently successful with primitive motions, future work aims to address directional ambiguity (e.g., "the ball rolled") using Monte-Carlo methods to determine prototypical values. Additionally, the researchers are developing quantitative evaluation frameworks using human feedback to further refine the system's accuracy in representing complex human actions.

Summary updated

Abstract

This paper details the technical functionality of VoxSim, a system for generating three-dimensional visual simulations of natural language motion expressions. We use a rich formal model of events and their participants to generate simulations that satisfy the minimal constraints entailed by an utterance and its minimal model, relying on real-world semantic knowledge of physical objects and motion events. This paper outlines technical considerations of such a system, and discusses the implementation of the aforementioned semantic models as well as VoxSim’s suitability as a platform for examining linguistic and spatial reasoning questions.
BibTeX
@inproceedings{krishnaswamy2016multimodal,
  author = {Nikhil Krishnaswamy and James Pustejovsky},
  title = {{Multimodal Semantic Simulations of Linguistically Underspecified Motion Events}},
  booktitle = {Spatial Cognition},
  year = {2016},
  url = {https://openreview.net/forum?id=tE1k6lb9F8},
}
Figure