Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering

Ahjeong Seo, Gi-Cheon Kang, Joonhan Park, Byoung-Tak Zhang

TL;DR

This paper introduces Motion-Appearance Synergistic Networks (MASN), a novel architecture for video question answering (Video QA) that effectively fuses motion and appearance information. By employing specialized modules and a question-guided fusion mechanism, MASN achieves state-of-the-art performance on challenging Video QA benchmarks, demonstrating the importance of synergistically combining visual modalities for understanding video content.

What Problem Does This Paper Address?

Video Question Answering (Video QA) is a complex task that requires AI systems to understand both the visual content of a video and a natural language question about it, then provide an accurate answer. This task presents several challenges: (1) interpreting the diverse intentions behind questions (e.g., asking about object appearance, actions, or causality), (2) capturing various visual elements within a video (objects, actions, temporal dynamics), and (3) grounding language and vision information across modalities. Previous approaches often struggled with effectively integrating motion and appearance cues, with some relying on object detection features that lack temporal modeling or using motion features that are limited in their reasoning capabilities, especially when dealing with long-term dependencies in videos.

Methods

The proposed Motion-Appearance Synergistic Networks (MASN) tackle these challenges with a three-module architecture:

  • Motion Module & Appearance Module: These parallel modules are designed to extract rich, cross-modal representations. They share a similar structure, but differ in their input visual features. The Appearance Module uses features from ResNet (for object appearance), while the Motion Module uses features from I3D (for action and motion). Both modules first construct object graphs using Graph Convolutional Networks (GCNs) to capture spatio-temporal relationships between objects. Then, a Vision-Question (VQ) interaction submodule performs cross-modal grounding by calculating correlations between the object graph features and the question features. This results in separate motion-specific and appearance-specific representations.
  • Motion-Appearance Fusion Module: This is the core of MASN. It takes the outputs from the Motion and Appearance modules and fuses them in a question-guided manner. It first uses attention mechanisms to create three types of attended features: appearance-centered, motion-centered, and all-mixed. These are then combined using a question-guided fusion strategy, where the model learns to dynamically weigh the importance of appearance and motion information based on the specific question. This allows the model to "attend what it needs."

Datasets

The paper evaluates MASN on three widely-used Video QA datasets:

  • TGIF-QA: A large-scale dataset with 165K QA pairs from animated GIFs, featuring tasks like Count, Action, State Transition, and FrameQA.
  • MSVD-QA & MSRVTT-QA: Datasets automatically generated from video descriptions, consisting of approximately 1,970 video clips and 50K QA pairs (MSVD-QA) and 243K QA pairs (MSRVTT-QA). These datasets primarily focus on "wh" questions.

Key Insights and Ideas

  • Synergy between Motion and Appearance: The paper strongly advocates for the synergistic combination of motion and appearance features. Simply concatenating them is less effective than a fusion mechanism that intelligently weighs their contributions based on the question.
  • Question-Guided Attention: The core innovation lies in the question-guided fusion. By learning to dynamically adjust the attention weights for appearance and motion features based on the question, MASN can prioritize the most relevant visual information for answering specific queries.
  • Object Graphs for Spatio-Temporal Reasoning: Using GCNs to build object graphs allows the model to explicitly model relationships between objects in a spatio-temporal context, which is crucial for understanding complex video events.
  • Complementary Roles of Modalities: The ablation studies suggest that motion and appearance features play complementary roles. While appearance features are strong for static details, motion features are essential for understanding actions and temporal dynamics.

Key Results

  • State-of-the-Art Performance: MASN achieves new state-of-the-art results on the TGIF-QA and MSVD-QA datasets.
  • On TGIF-QA, MASN outperforms previous methods on Count, Action, and Trans. tasks, achieving 84.4% on Action and 87.4% on Trans.
  • On MSVD-QA, MASN achieves 38.0% accuracy, outperforming the best baseline by approximately 2%.
  • Ablation Study Effectiveness: Ablation studies demonstrate the effectiveness of each component. The full MASN architecture (with both motion and appearance modules and the fusion module) significantly outperforms models using only one modality or simpler fusion methods.
  • Qualitative Analysis: Visualizations of attention maps show that MASN's attention mechanisms effectively focus on relevant objects and actions, and that the question-guided fusion dynamically shifts focus between appearance and motion cues depending on the question type.

Limitations and Future Work

  • Limitations: The paper notes that while the dense attention maps in the fusion module contribute to performance, they might lead to inconsistencies in certain tasks (e.g., Count) compared to others that rely more on motion. The authors also mention that the model might struggle with tasks where motion information is less prominent.
  • Future Work: The authors suggest that further exploration of attention mechanisms and fusion strategies could lead to even better performance. The current architecture could potentially be extended to handle more complex video QA scenarios or other vision-language tasks.

Summary updated

Abstract

Video Question Answering is a task which requires an AI agent to answer questions grounded in video. This task entails three key challenges: (1) understand the intention of various questions, (2) capturing various elements of the input video (e.g., object, action, causality), and (3) cross-modal grounding between language and vision information. We propose Motion-Appearance Synergistic Networks (MASN), which embed two cross-modal features grounded on motion and appearance information and selectively utilize them depending on the question's intentions. MASN consists of a motion module, an appearance module, and a motion-appearance fusion module. The motion module computes the action-oriented cross-modal joint representations, while the appearance module focuses on the appearance aspect of the input video. Finally, the motion-appearance fusion module takes each output of the motion module and the appearance module as input, and performs question-guided fusion. As a result, MASN achieves new state-of-the-art performance on the TGIF-QA and MSVD-QA datasets. We also conduct qualitative analysis by visualizing the inference results of MASN. The code is available at https://github.com/ahjeongseo/MASN-pytorch.
BibTeX
@inproceedings{seo2021attend,
  author = {Ahjeong Seo and Gi-Cheon Kang and Joonhan Park and Byoung-Tak Zhang},
  title = {{Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.481},
}
Figure