Abstract
This paper introduces a novel Japanese video-and-language dataset designed to facilitate multimodal logical inference. Unlike existing datasets that focus on static images or simple action labels, this work emphasizes "intentional" and "aspectual" actions—such as trying to perform an action or continuing an activity. By mapping these dynamic human movements to First-Order Logic (FOL) across temporal scenes, the dataset provides a structured framework for AI to reason about complex, non-binary temporal scenarios.
Introduction
Visual-Textual Entailment (VTE) typically evaluates whether a sentence holds true against a visual premise. While static images suffice for simple state descriptions, they fail to capture the nuances of dynamic video, such as an action being attempted but not completed. Understanding such scenarios requires handling linguistic phenomena like negation and quantification within a temporal context. This research addresses the lack of structured logical representations in current datasets by creating a mapping between human actions and First-Order Logic.
Methodology
The proposed framework represents video information as a sequence of time-ordered "scenes" ($S$).
- Triplet Representation: Actions are annotated as
(subject, predicate, object)triplets, such as(person, try to eat, food). - Temporal Logic Extension: Standard FOL is extended to track attribute and relation changes across scenes. This allows the system to track how attributes and relations change from one scene to the next.
- Logical Translation: These triplets are converted into FOL formulas that assert relationships within specific temporal slices.
- Free-form Annotation: Annotators used natural Japanese predicates rather than fixed categories, capturing a broader range of expressions.
Dataset Characteristics
The researchers utilized 200 videos from the Charades test set, focusing on multi-person interactions. The resulting benchmark contains 5,554 action labels and 1,942 unique categories. This free-form approach significantly increased vocabulary richness compared to the original Charades dataset (157 categories) and showed higher label density than ActionGenome or STAIR Actions. Notably, 65% of the action labels in the dataset appear only once, reflecting the "long tail" of natural human activity.
Results and Analysis
The dataset achieved high granularity, averaging 27.77 action labels per video. A key contribution is the successful capture of specialized predicates that are often overlooked in traditional action recognition, specifically those related to utterance, intention, and aspect.
The focus on intentionality (e.g., "trying to...") allows for logical inference in cases where actions are incomplete—a common real-world occurrence that static images cannot capture. By grounding video actions in FOL, the authors provide a path toward "explainable" multimodal AI, where a system can provide a logical proof for its entailment decisions.
Conclusion
While the dataset is currently limited in scale (200 videos) and the frequency of complex aspectual phrases is relatively low (1.13% combined), it establishes a foundation for explainable multimodal AI. Future work will look toward scaling the annotation process using automated models and longer video sequences to capture more complex, long-term temporal changes.
Summary updated