Temporal Pyramid Transformer with Multimodal Interaction for Video Question Answering

Min Peng, Chongyang Wang, Yuan Gao, Yu Shi, Xiang-Dong ZhouarXiv 2021 · OpenReview

TL;DR

This paper introduces the Temporal Pyramid Transformer (TPT), a novel architecture for Video Question Answering (VideoQA) that effectively leverages visual information across multiple temporal scales. By constructing a "temporal pyramid" of video features and employing specialized Transformer modules for question understanding and visual inference, TPT achieves state-of-the-art results on several benchmark datasets.

What Problem Does This Paper Address?

Video Question Answering (VideoQA) is a complex task that requires understanding both the visual content of a video and the nuances of a natural language question to predict an answer. Existing methods often struggle to fully exploit the rich, multi-scale temporal dynamics present in videos and to deeply integrate the question's semantics with the visual information. Specifically, the paper highlights that:

  • Underutilization of Temporal Scales: Many approaches don't effectively leverage appearance and motion information across different temporal granularities (e.g., short, rapid movements vs. longer, broader actions).
  • Ignored Question-Video Interaction: The interaction between the question's semantics and the visual information is often treated superficially, hindering a deep understanding of what visual cues are relevant to the question.
  • Hierarchical Nature of Visual Clues: The visual information needed to answer a question can be hierarchical. For instance, identifying "pandas" might require a coarse view, while understanding "play on" might need finer, sequential details.

Methods

The proposed Temporal Pyramid Transformer (TPT) model addresses these challenges with two core modules:

  • Temporal Pyramid Construction: The input video is first processed to create a "temporal pyramid." This involves segmenting the video at different temporal resolutions (levels). At each level, appearance and motion features are extracted from groups of frames. This creates a multi-scale representation of the video.
  • Question-Specific Transformer (QT) Module: This module focuses on understanding the question in the context of the video. It uses a Transformer architecture to build "question semantics" by looking at the co-occurrence of question words with visual content across the different levels of the temporal pyramid. This is a coarse-to-fine process, aiming to deeply understand the question's relevance to the video.
  • Visual Inference (VI) Module: Guided by the question semantics generated by the QT module, this module infers the relevant visual clues. It performs a local-to-global search across the temporal pyramid, leveraging multi-level interactions between the question and video features. This module aims to extract the specific visual information needed for answering.

Both QT and VI modules utilize a multimodal attention mechanism to fuse question and video features and residual connections to pass information between different levels of their respective pyramids, enabling a more robust and hierarchical processing.

Datasets

The paper evaluates TPT on three widely-used VideoQA benchmark datasets:

  • TGIF-QA: A large-scale dataset with animated GIFs, featuring tasks like Action, Transition, FrameQA, and Count.
  • MSVD-QA: A dataset of short video clips with open-ended questions categorized by "what, who, how, when, and where."
  • MSRVTT-QA: A dataset with longer videos (10-30 seconds) and more complex scenarios, also featuring open-ended questions.

Key Insights and Ideas

  • Temporal Pyramid is Crucial: Removing the temporal pyramid and using features from a single level significantly degrades performance, highlighting the importance of multi-scale temporal information for VideoQA.
  • QT Module Enhances Understanding: Disabling the QT module (which specifically builds question semantics from video context) leads to performance drops across all tasks, underscoring the value of explicitly modeling question-video semantic interaction.
  • Task-Specific Pyramid Depth: The optimal number of levels in the temporal pyramid can vary by task. For instance, FrameQA benefits less from deeper pyramids, while Count tasks improve with more levels, suggesting a need to balance global and local information.
  • Hierarchical Clues: The paper visually demonstrates how different parts of a question might require visual information from different temporal scales (e.g., "pandas" at a coarse level, "play on" at a finer level).

Key Results

The TPT model achieves state-of-the-art performance on several tasks across the evaluated datasets:

  • TGIF-QA: TPT outperforms previous methods on the Action and Count tasks, achieving the best performance with 76.6% (Action), 81.6% (Trans.), 57.8% (FrameQA), and 3.63 (Count).
  • MSVD-QA & MSRVTT-QA: TPT achieves the best results on both datasets, with 38.5% accuracy on MSVD-QA and 37.7% on MSRVTT-QA.

Qualitative examples show that while TPT can sometimes misinterpret fine-grained details (e.g., mistaking a lemon for a toy), it generally performs well in identifying correct answers, as demonstrated by the "three kids performing a song" example.

Limitations and Future Work

  • Limitations:
    • The paper notes that using a standard ResNet for spatial feature extraction might limit the capture of fine-grained spatial details, potentially impacting performance on tasks requiring such precision.
  • The performance on the "Trans." task in TGIF-QA is slightly lower than some graph-based methods, suggesting that more advanced spatial-temporal reasoning might be beneficial.
  • Future Work:
    • Improve the appearance feature extraction to better capture small targets within images.
  • Explore bidirectional temporal processing strategies to better capture informative events across different temporal scales.
  • Investigate adjusting the temporal pyramid's level count based on specific task requirements to optimize the balance between global and local visual information.

Summary updated

Abstract

Video question answering (VideoQA) is challenging given its multimodal combination of visual understanding and natural language understanding. While existing approaches seldom leverage the appearance-motion information in the video at multiple temporal scales, the interaction between the question and the visual information for textual semantics extraction is frequently ignored. Targeting these issues, this paper proposes a novel Temporal Pyramid Transformer (TPT) model with multimodal interaction for VideoQA. The TPT model comprises two modules, namely Question-specific Transformer (QT) and Visual Inference (VI). Given the temporal pyramid constructed from a video, QT builds the question semantics from the coarse-to-fine multimodal co-occurrence between each word and the visual content. Under the guidance of such question-specific semantics, VI infers the visual clues from the local-to-global multi-level interactions between the question and the video. Within each module, we introduce a multimodal attention mechanism to aid the extraction of question-video interactions, with residual connections adopted for the information passing across different levels. Through extensive experiments on three VideoQA datasets, we demonstrate better performances of the proposed method in comparison with the state-of-the-arts.
BibTeX
@misc{peng2021temporal,
  author = {Min Peng and Chongyang Wang and Yuan Gao and Yu Shi and Xiang-Dong Zhou},
  title = {{Temporal Pyramid Transformer with Multimodal Interaction for Video Question Answering}},
  year = {2021},
  eprint = {2109.04735},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.04735},
}
Figure