Compositional Attention Networks With Two-Stream Fusion for Video Question Answering
Lists
Only you see this
Abstract
Given a video, Video Question Answering (VideoQA) aims at answering arbitrary free-form questions about the video content in natural language. A successful VideoQA framework usually has the following two key components: 1) a discriminative video encoder that learns the effective video representation to maintain as much information as possible about the video and 2) a question-guided decoder that learns to select the most related features to perform spatiotemporal reasoning, as well as outputs the correct answer. We propose compositional attention networks (CAN) with two-stream fusion for VideoQA tasks. For the encoder, we sample video snippets using a two-stream mechanism (i.e., a uniform sampling stream and an action pooling stream) and extract a sequence of visual features for each stream to represent the video semantics with implementation. For the decoder, we propose a compositional attention module to integrate the two-stream features with the attention mechanism. The compositional attention module is the core of CAN and can be seen as a modular combination of a unified attention block. With different fusion strategies, we devise five compositional attention module variants. We evaluate our approach on one long-term VideoQA dataset, ActivityNet-QA, and two short-term VideoQA datasets, MSRVTT-QA and MSVD-QA. Our CAN model achieves new state-of-the-art results on all the datasets.
BibTeX
@article{yu2020compositional,
author = {Ting Yu and Jun Yu and Zhou Yu and Dacheng Tao},
title = {{Compositional Attention Networks With Two-Stream Fusion for Video Question Answering}},
journal = {IEEE Trans. Image Process.},
year = {2020},
doi = {10.1109/tip.2019.2940677},
}Loading related content...