Long-Form Video Question Answering via Dynamic Hierarchical Reinforced Networks

Zhou Zhao, Zhu Zhang, Shuwen Xiao, Zhenxin Xiao, Xiaohui Yan, Jun Yu, Deng Cai, Fei Wu

Abstract

Open-ended long-form video question answering is a challenging task in visual information retrieval, which automatically generates a natural language answer from the referenced long-form video contents according to a given question. However, the existing works mainly focus on short-form video question answering, due to the lack of modeling semantic representations from long-form video contents. In this paper, we introduce a dynamic hierarchical reinforced network for open-ended long-form video question answering, which employs an encoder-decoder architecture with a dynamic hierarchical encoder and a reinforced decoder. Concretely, we first propose a frame-level dynamic long-short term memory (LSTM) network with binary segmentation gate to learn frame-level semantic representations according to the given question. We then develop a segment-level highway LSTM network with a question-aware highway gate for segment-level semantic modeling. Furthermore, we devise the reinforced decoder with a hierarchical attention mechanism to generate natural language answers. We construct a large-scale long-form video question answering dataset. The extensive experiments on the long-form dataset and another public short-form dataset show the effectiveness of our method.
BibTeX
@article{zhao2019long,
  author = {Zhou Zhao and Zhu Zhang and Shuwen Xiao and Zhenxin Xiao and Xiaohui Yan and Jun Yu and Deng Cai and Fei Wu},
  title = {{Long-Form Video Question Answering via Dynamic Hierarchical Reinforced Networks}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  doi = {10.1109/tip.2019.2922062},
}

Loading related content...

Figure