Introduction
In Visual Question Answering (VQA), the mechanism of multimodal fusion is a core research challenge. While early models relied on simple vector concatenation or addition, Bilinear Pooling (BLP) emerged as a breakthrough in Image-QA. By utilizing outer products, BLP allows for expressive, all-to-all interactions between visual and textual feature vectors. However, its efficacy in the temporal, high-dimensional domain of Video-QA has remained largely unverified. This study systematically evaluates various BLP techniques across established Video-QA architectures to determine if the successes in static image analysis translate to video.
Methodology
The researchers integrated BLP into two primary Video-QA frameworks to assess its impact:
- TVQA Baseline and Dual-Stream Models: The standard concatenation layers in the TVQA model were replaced with several BLP variants, including Multi-modal Compact Bilinear pooling (MCB), Low-rank Bilinear pooling (MLB), Multi-modal Factorized Bilinear pooling (MFB), and BLOCK. A restructured "Dual-Stream" version was also developed to better accommodate joint representations.
- HME-VideoQA: BLP was integrated into the fusion unit of the Heterogeneous-Memory-Enhanced (HME) model, which utilizes external memory to track evolving video and question features.
Experimental Setup
The study utilized four diverse benchmarks to ensure a robust evaluation:
- TVQA: Compositional questions based on long clips from TV shows.
- TGIF-QA: Action counting and state transition identification in animated GIFs.
- MSVD-QA: Short clips with "5W" (who, what, where, etc.) questions.
- Ego-VQA: Challenging first-person perspective videos.
Key Results and Analysis
Contrary to expectations, BLP did not serve as a "plug-and-play" performance booster. In the TVQA dataset, replacing concatenation with BLP led to an accuracy decrease of 0.3% to 5%. The Dual-Stream model performed even worse, with a 5-10% drop. While the HME model saw a modest +2.87% boost on TGIF-Action using MCB, gains on other datasets were negligible. Notably, the BLP-integrated HME model failed to converge on the Ego-VQA dataset, indicating high sensitivity to hyperparameters.
Several factors contribute to these findings:
- The Joint Representation Trap: Forcing modalities into a single joint space may obscure modality-specific information, which is problematic in Video-QA where "unimodal bias" (answering from text alone) is common.
- Attention vs. Fusion: Sophisticated temporal attention appears more critical than the fusion operation itself. Without effective filtering, BLP pools irrelevant "noise."
- Hardware Constraints: The high memory requirements of bilinear representations limited hidden dimensions to ~1600, far below the ~16,000 often required for BLP to outperform simpler methods in Image-QA.
Conclusion
The study suggests that for Video-QA, prioritizing temporal attention and maintaining discrete feature maps may be more productive than complex global vector fusion. Future work should explore non-linear extensions of BLP and higher-order interactions that can simultaneously manage multimodal and temporal relations.
Summary updated