Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference

Riko Suzuki, Hitomi Yanaka, Koji Mineshima, Daisuke BekkiarXiv 2021 · arXiv

Abstract

This paper introduces a novel Japanese video-and-language dataset designed to facilitate multimodal logical inference. Unlike existing datasets that focus on static images or simple action labels, this work emphasizes "intentional" and "aspectual" actions—such as trying to perform an action or continuing an activity. By mapping these dynamic human movements to First-Order Logic (FOL) across temporal scenes, the dataset provides a structured framework for AI to reason about complex, non-binary temporal scenarios.

Introduction

Visual-Textual Entailment (VTE) typically evaluates whether a sentence holds true against a visual premise. While static images suffice for simple state descriptions, they fail to capture the nuances of dynamic video, such as an action being attempted but not completed. Understanding such scenarios requires handling linguistic phenomena like negation and quantification within a temporal context. This research addresses the lack of structured logical representations in current datasets by creating a mapping between human actions and First-Order Logic.

Methodology

The proposed framework represents video information as a sequence of time-ordered "scenes" ($S$).

  1. Triplet Representation: Actions are annotated as (subject, predicate, object) triplets, such as (person, try to eat, food).
  2. Temporal Logic Extension: Standard FOL is extended to track attribute and relation changes across scenes. This allows the system to track how attributes and relations change from one scene to the next.
  3. Logical Translation: These triplets are converted into FOL formulas that assert relationships within specific temporal slices.
  4. Free-form Annotation: Annotators used natural Japanese predicates rather than fixed categories, capturing a broader range of expressions.
Temporal logic representation of actions across scenes

Dataset Characteristics

The researchers utilized 200 videos from the Charades test set, focusing on multi-person interactions. The resulting benchmark contains 5,554 action labels and 1,942 unique categories. This free-form approach significantly increased vocabulary richness compared to the original Charades dataset (157 categories) and showed higher label density than ActionGenome or STAIR Actions. Notably, 65% of the action labels in the dataset appear only once, reflecting the "long tail" of natural human activity.

Results and Analysis

The dataset achieved high granularity, averaging 27.77 action labels per video. A key contribution is the successful capture of specialized predicates that are often overlooked in traditional action recognition, specifically those related to utterance, intention, and aspect.

Frequency and examples of specialized predicates captured in the dataset

The focus on intentionality (e.g., "trying to...") allows for logical inference in cases where actions are incomplete—a common real-world occurrence that static images cannot capture. By grounding video actions in FOL, the authors provide a path toward "explainable" multimodal AI, where a system can provide a logical proof for its entailment decisions.

Conclusion

While the dataset is currently limited in scale (200 videos) and the frequency of complex aspectual phrases is relatively low (1.13% combined), it establishes a foundation for explainable multimodal AI. Future work will look toward scaling the annotation process using automated models and longer video sequences to capture more complex, long-term temporal changes.

Summary updated

Abstract

This paper introduces a new video-and-language dataset with human actions for multimodal logical inference, which focuses on intentional and aspectual expressions that describe dynamic human actions. The dataset consists of 200 videos, 5,554 action labels, and 1,942 action triplets of the form <subject, predicate, object> that can be translated into logical semantic representations. The dataset is expected to be useful for evaluating multimodal inference systems between videos and semantically complicated sentences including negation and quantification.
BibTeX
@misc{suzuki2021building,
  author = {Riko Suzuki and Hitomi Yanaka and Koji Mineshima and Daisuke Bekki},
  title = {{Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference}},
  year = {2021},
  eprint = {2106.14137},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.14137},
}
Figure