Grounding the Lexical Semantics of Verbs in Visual Perception using Force Dynamics and Event Logic

Introduction

Recognizing human actions from video—such as "picking up" or "stacking" an object—is a foundational challenge in computer vision. Traditional approaches often rely on tracking "motion profiles," which analyze the continuous trajectories, velocities, and accelerations of objects. However, these methods are inherently brittle; a "pick up" action looks drastically different depending on the speed, direction, or background clutter. This paper introduces LEONARD, a system that shifts the focus from raw motion to the underlying physical relationships—force dynamics—and logical state changes.

Methodology: Force Dynamics and Event Logic

LEONARD operates by inferring invisible physical properties from video sequences. Instead of just tracking pixels, it reconstructs a model of the scene based on three core concepts:

1. Force Dynamics: The system determines if an object is grounded, attached to another (like a hand), or supporting another object.
2. Event Logic: Verbs are defined as logical transitions between these physical states. For instance, a PICKUP event is not defined by an upward trajectory but by a sequence: Object A supports Object B $\rightarrow$ Hand attaches to Object B $\rightarrow$ Hand supports Object B.
3. Spanning Intervals: To handle "liquid events" (states that remain true over any sub-interval), the system employs a mathematical representation called spanning intervals. This allows for efficient computation of logical rules over continuous time.

Expand

Source, p. 2

Experimental Evaluation

The system was tested on a dataset of 35 video sequences (320x240 at 30fps) involving a human hand manipulating colored blocks. The dataset covered seven event types: pick up, put down, stack, unstack, move, assemble, and disassemble.

Key Results

LEONARD demonstrated remarkable precision, achieving zero false positives across the test set. It successfully ignored random motions that did not meet the logical criteria for a specific verb. While it correctly classified most events, six false negatives occurred in complex, multi-stage actions like ASSEMBLE, where a constituent sub-event (like STACK) was missed.

Discussion and Insights

The primary insight of this work is that the essence of spatial verbs is causal and physical rather than purely geometric. By focusing on support and attachment, the system gains several advantages:

• Invariance to Trajectory: A "pick up" is recognized regardless of the hand's approach path.
• Hierarchical Understanding: Complex actions like MOVE are naturally understood as a sequence of simpler actions (PICKUP followed by PUTDOWN).
• Robustness to Clutter: Objects not involved in the causal chain of support are effectively ignored, preventing background noise from triggering false detections.

Conclusion and Future Work

While LEONARD proves the power of grounding semantics in physics, it currently relies on hand-coded physical rules and operates in a simplified "blocks world." Future research aims to automate the learning of these physical laws and lexical definitions directly from visual data, moving toward a more flexible system capable of handling the nuances and metaphors present in natural language.