Human-centric Spatio-Temporal Video Grounding With Visual Transformers

Zongheng Tang, Yue Liao, Si Liu, Guanbin Li, Xiaojie Jin, Hongxu Jiang, Qian Yu, Dong Xu

TL;DR

This paper introduces "Human-centric Spatio-Temporal Video Grounding" (HC-STVG), a new task focused on precisely locating a specific person within untrimmed videos using natural language descriptions. They propose a novel baseline method, STGVT, leveraging Visual Transformers, and release a new dataset to facilitate research in this area, demonstrating superior performance over existing approaches.

What Problem Does This Paper Address?

Current video grounding tasks often focus on localizing general objects or temporal events. However, in real-world applications like surveillance or healthcare, the critical need is to track a specific person performing actions over time within potentially very long, untrimmed videos. Existing methods struggle with this because they either lack the temporal dimension, don't focus on humans, or can't handle the complexity of multi-person scenes and nuanced descriptions. This gap leaves a need for a task that specifically addresses human-centric spatio-temporal localization, requiring both precise spatial bounding boxes and accurate temporal segmentation of a target individual.

Methods

The authors propose Spatio-Temporal Grounding with Visual Transformers (STGVT), a novel baseline method designed for the HC-STVG task. STGVT operates in four main stages:

  1. Tube Proposals Generation: It first generates potential spatio-temporal "tube" proposals by linking bounding boxes detected in consecutive frames. This stage uses a similarity score that combines Intersection over Union (IoU) and cosine similarity of features to link boxes across frames.
  2. Cross-modal Representation: A Visual Transformer is employed to process pairs of tube proposals and query sentences. This transformer models the interaction between visual (from the video tubes) and textual (from the query) modalities, extracting both global and frame-level features.
  3. Tube-description Matching: A classifier uses the global features from the Visual Transformer to predict a matching score for each tube-sentence pair. The tube with the highest score is selected.
  4. Tube Trimming: To refine the selected tube, a trimming module is used. This module employs a classification subnet to determine frame relevance and a boundary regression subnet to predict temporal offsets, ultimately producing the final spatio-temporal tube.

Datasets

The paper introduces a new benchmark dataset called HC-STVG.

  • HC-STVG Dataset: Contains 5,660 video-sentence pairs.
  • Videos are 20 seconds long and feature complex multi-person scenes (over 57% have more than three people).
  • Sentences are natural language descriptions with an average of 17.25 words, rich in actions and human-object interactions.
  • Provides spatio-temporal annotations for the target person.

The authors also evaluate their method on the VidSTG dataset for comparison, noting that VidSTG is not human-centric but provides spatio-temporal annotations.

Key Insights and Ideas

  • Human-centricity is Crucial: Focusing specifically on humans for spatio-temporal grounding is a significant step, as humans are often the primary subjects of interest in surveillance and healthcare scenarios.
  • Visual Transformers for Cross-Modal Reasoning: The paper highlights the effectiveness of Visual Transformers in jointly processing visual and textual information for complex spatio-temporal tasks, enabling better understanding of actions and relations.
  • The Challenge of Untrimmed Videos: The HC-STVG task's focus on untrimmed videos adds a layer of difficulty, requiring robust methods to sift through long sequences and pinpoint specific events.
  • Importance of Fine-grained Reasoning: Accurately grounding descriptions involving actions and interactions necessitates fine-grained spatio-temporal reasoning, which the proposed STGVT method aims to achieve.

Key Results

  • The proposed STGVT method significantly outperforms existing baseline methods on the newly introduced HC-STVG dataset.
  • On the HC-STVG dataset, STGVT achieves an m_vIoU of 18.15%, outperforming baselines by a significant margin. It also surpasses baselines in [email protected] and [email protected].
  • Experiments on the VidSTG dataset also show STGVT's effectiveness, achieving an m_vIoU of 21.62%.
  • Ablation studies demonstrate the effectiveness of both the tube-description matching module and the tube trimming module. The full STGVT method (with trimming) significantly outperforms the version without trimming.
  • Qualitative analysis shows STGVT achieving more accurate spatial and temporal localization compared to baselines in several examples.

Limitations and Future Work

  • Pretraining Strategy: While pretraining the Visual Transformer on Conceptual Captions improves performance, training from scratch still yields competitive results, suggesting the architecture's inherent strength. However, the paper notes that using pretraining can improve performance.
  • Dataset Diversity: The paper mentions the dataset is designed to be diverse, but further expansion and exploration of more complex scenarios could be beneficial.
  • Future Work: The authors suggest that future work could explore more advanced transformer architectures and potentially incorporate audio cues for even richer video understanding. The HC-STVG task itself opens avenues for developing more sophisticated models for human-centric video analysis.

Summary updated

Abstract

In this work, we introduce a novel task - Humancentric Spatio-Temporal Video Grounding (HC-STVG). Unlike the existing referring expression tasks in images or videos, by focusing on humans, HC-STVG aims to localize a spatiotemporal tube of the target person from an untrimmed video based on a given textural description. This task is useful, especially for healthcare and security-related applications, where the surveillance videos can be extremely long but only a specific person during a specific period of time is concerned. HC-STVG is a video grounding task that requires both spatial (where) and temporal (when) localization. Unfortunately, the existing grounding methods cannot handle this task well. We tackle this task by proposing an effective baseline method named Spatio-Temporal Grounding with Visual Transformers (STGVT), which utilizes Visual Transformers to extract cross-modal representations for video-sentence matching and temporal localization. To facilitate this task, we also contribute an HC-STVG dataset consisting of 5,660 video-sentence pairs on complex multi-person scenes. Specifically, each video lasts for 20 seconds, pairing with a natural query sentence with an average of 17.25 words. Extensive experiments are conducted on this dataset, demonstrating the newly-proposed method outperforms the existing baseline methods.
BibTeX
@article{tang2020human,
  author = {Zongheng Tang and Yue Liao and Si Liu and Guanbin Li and Xiaojie Jin and Hongxu Jiang and Qian Yu and Dong Xu},
  title = {{Human-centric Spatio-Temporal Video Grounding With Visual Transformers}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2020},
  url = {https://openreview.net/forum?id=4NUXEuF3KR2},
}
Figure