Identifying Visible Actions in Lifestyle Vlogs

Oana Ignat, Laura Burdick, Jia Deng, Rada Mihalcea

Introduction

Traditional action recognition often relies on explicit data gathering, where researchers define a set of labels and find matching videos. While precise, this approach often fails to capture the messy, diverse ways humans describe and perform actions in real-world settings. This paper addresses the inverse problem: implicit data gathering. By analyzing lifestyle vlogs—where creators narrate their daily routines—the researchers aim to determine if a narrated action is actually visible on screen, a challenge known as "grounding."

Methodology

The authors propose a multimodal neural architecture designed to fuse linguistic and visual information to identify grounded actions. The pipeline involves several stages of filtering and feature extraction to bridge the gap between spoken words and visual frames.

The proposed pipeline for grounding actions, from transcript filtering to motion filtering.

Multimodal Features

  • Linguistic Features: The model uses ELMo and GloVe for semantic representation. A key innovation is the inclusion of a "concreteness" score (1–5) for words; high concreteness (e.g., "water") often correlates with visibility, whereas abstract terms (e.g., "thoughts") do not.
  • Visual Features: Frame-level features are extracted via Inception V3, while spatiotemporal features are captured using C3D to account for motion.
  • Architecture: These features are processed through an LSTM and concatenated with linguistic data (including Part-of-Speech tags and context) before a feed-forward network calculates the probability of visibility.

New Lifestyle Vlog Dataset

The study introduces a novel dataset curated from YouTube, consisting of 1,268 short video clips (miniclips) derived from 177 full-length videos. Unlike standard datasets with fixed labels, this dataset uses "post-defined" labels—the actual words used by vloggers—resulting in a richer vocabulary of actions.

Examples of segmented miniclips used for manual annotation of visible actions.

Of the 14,769 actions extracted from transcripts, 4,340 were manually labeled as "visible" by human annotators. This dataset captures the "long tail" of human activity, such as the subtle difference between "popping" something into a freezer versus "sticking" it in.

Key Insights and Results

  • Text as a Strong Prior: Linguistic models alone (ELMo) achieved an impressive F1 score of 0.813, suggesting that the structure of narration provides significant clues about visibility.
  • Concreteness is Critical: There is a strong correlation between the physical nature of a verb or noun and its likelihood of being visible on screen.
  • Multimodal Superiority: The combined model outperformed all baselines, achieving a top F1 score of 0.827. In contrast, simple object detection (YOLO9000) struggled (F1: 0.520), proving that seeing an object is not enough to confirm an action is occurring.

Limitations and Future Work

While successful in the vlog domain, the model's performance in less tightly coupled media (like news or movies) is unknown. Future research may focus on reducing noise from auto-generated transcripts and improving the temporal alignment between speech and action, which often features a lag of several seconds.

Summary updated

BibTeX
@inproceedings{ignat2019identifying,
  author = {Oana Ignat and Laura Burdick and Jia Deng and Rada Mihalcea},
  title = {{Identifying Visible Actions in Lifestyle Vlogs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1643},
}
Figure