Trying Bilinear Pooling in Video-QA

Thomas Winterbottom, Sarah Xiao, Alistair McLean, Noura Al MoubayedarXiv 2020 · arXiv

Introduction

In Visual Question Answering (VQA), the mechanism of multimodal fusion is a core research challenge. While early models relied on simple vector concatenation or addition, Bilinear Pooling (BLP) emerged as a breakthrough in Image-QA. By utilizing outer products, BLP allows for expressive, all-to-all interactions between visual and textual feature vectors. However, its efficacy in the temporal, high-dimensional domain of Video-QA has remained largely unverified. This study systematically evaluates various BLP techniques across established Video-QA architectures to determine if the successes in static image analysis translate to video.

Methodology

The researchers integrated BLP into two primary Video-QA frameworks to assess its impact:

  1. TVQA Baseline and Dual-Stream Models: The standard concatenation layers in the TVQA model were replaced with several BLP variants, including Multi-modal Compact Bilinear pooling (MCB), Low-rank Bilinear pooling (MLB), Multi-modal Factorized Bilinear pooling (MFB), and BLOCK. A restructured "Dual-Stream" version was also developed to better accommodate joint representations.
The TVQA baseline architecture featuring various stream processors for subtitles, visual concepts, regional features, and image features.
  1. HME-VideoQA: BLP was integrated into the fusion unit of the Heterogeneous-Memory-Enhanced (HME) model, which utilizes external memory to track evolving video and question features.
The HME-VideoQA architecture, highlighting the visual and question memory modules and the multimodal fusion unit.

Experimental Setup

The study utilized four diverse benchmarks to ensure a robust evaluation:

  • TVQA: Compositional questions based on long clips from TV shows.
  • TGIF-QA: Action counting and state transition identification in animated GIFs.
  • MSVD-QA: Short clips with "5W" (who, what, where, etc.) questions.
  • Ego-VQA: Challenging first-person perspective videos.

Key Results and Analysis

Contrary to expectations, BLP did not serve as a "plug-and-play" performance booster. In the TVQA dataset, replacing concatenation with BLP led to an accuracy decrease of 0.3% to 5%. The Dual-Stream model performed even worse, with a 5-10% drop. While the HME model saw a modest +2.87% boost on TGIF-Action using MCB, gains on other datasets were negligible. Notably, the BLP-integrated HME model failed to converge on the Ego-VQA dataset, indicating high sensitivity to hyperparameters.

Several factors contribute to these findings:

  • The Joint Representation Trap: Forcing modalities into a single joint space may obscure modality-specific information, which is problematic in Video-QA where "unimodal bias" (answering from text alone) is common.
  • Attention vs. Fusion: Sophisticated temporal attention appears more critical than the fusion operation itself. Without effective filtering, BLP pools irrelevant "noise."
  • Hardware Constraints: The high memory requirements of bilinear representations limited hidden dimensions to ~1600, far below the ~16,000 often required for BLP to outperform simpler methods in Image-QA.

Conclusion

The study suggests that for Video-QA, prioritizing temporal attention and maintaining discrete feature maps may be more productive than complex global vector fusion. Future work should explore non-linear extensions of BLP and higher-order interactions that can simultaneously manage multimodal and temporal relations.

Summary updated

Abstract

Bilinear pooling (BLP) refers to a family of operations recently developed for fusing features from different modalities predominantly developed for VQA models. A bilinear (outer-product) expansion is thought to encourage models to learn interactions between two feature spaces and has experimentally outperformed `simpler' vector operations (concatenation and element-wise-addition/multiplication) on VQA benchmarks. Successive BLP techniques have yielded higher performance with lower computational expense and are often implemented alongside attention mechanisms. However, despite significant progress in VQA, BLP methods have not been widely applied to more recently explored video question answering (video-QA) tasks. In this paper, we begin to bridge this research gap by applying BLP techniques to various video-QA benchmarks, namely: TVQA, TGIF-QA, Ego-VQA and MSVD-QA. We share our results on the TVQA baseline model, and the recently proposed heterogeneous-memory-enchanced multimodal attention (HME) model. Our experiments include both simply replacing feature concatenation in the existing models with BLP, and a modified version of the TVQA baseline to accommodate BLP we name the `dual-stream' model. We find that our relatively simple integration of BLP does not increase, and mostly harms, performance on these video-QA benchmarks. Using recently proposed theoretical multimodal fusion taxonomies, we offer insight into why BLP-driven performance gain for video-QA benchmarks may be more difficult to achieve than in earlier VQA models. We suggest a few additional `best-practices' to consider when applying BLP to video-QA. We stress that video-QA models should carefully consider where the complex representational potential from BLP is actually needed to avoid computational expense on `redundant' fusion.
BibTeX
@misc{winterbottom2020trying,
  author = {Thomas Winterbottom and Sarah Xiao and Alistair McLean and Noura Al Moubayed},
  title = {{Trying Bilinear Pooling in Video-QA}},
  year = {2020},
  eprint = {2012.10285},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.10285},
}
Figure