Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering

Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, Dacheng Tao

Abstract

This paper introduces Multimodal Factorized High-order pooling (MFH), a novel method for fusing visual and textual information in Visual Question Answering (VQA). By cascading multiple factorized bilinear pooling blocks, the authors capture complex, high-order interactions between modalities. When combined with a dual "co-attention" mechanism and a distribution-based loss function, MFH achieved state-of-the-art results and secured the runner-up spot in the 2017 VQA Challenge.

Introduction

Visual Question Answering (VQA) is a complex multimodal task requiring a computer to interpret an image and answer a natural language question. This task is significantly more challenging than image captioning as it requires fine-grained reasoning and the ability to distinguish relevant information from noise.

The authors identify three major bottlenecks in existing VQA systems:

  1. Noisy Features: Global image features often contain irrelevant clutter, while questions contain linguistic filler.
  2. Ineffective Fusion: Standard methods like concatenation or addition fail to capture nuanced, non-linear correlations between pixels and words.
  3. Rigid Answer Prediction: Treating unique answers as separate, unrelated classes ignores semantic similarities (e.g., "puppy" vs. "little dog"), making training less efficient.

Methodology

The proposed architecture addresses these bottlenecks through three key innovations integrated into a unified deep learning framework.

Overall VQA pipeline integrating CNN for visual features, LSTM for textual features, and MFB/MFH for multimodal fusion.

Multimodal Factorized High-order Pooling (MFH)

While previous state-of-the-art methods utilized "bilinear pooling" (outer products) to fuse modalities, the approach was computationally expensive. The authors first propose MFB (Multimodal Factorized Bilinear pooling), which uses matrix factorization to achieve bilinear expressiveness with fewer parameters. They then extend this to MFH by cascading multiple MFB blocks. This allows the model to learn "high-order" relationships—essentially capturing interactions between interactions.

The MFH architecture illustrating the cascading of multiple MFB blocks to capture high-order interactions.

Critical to this fusion is normalization; the authors found that applying Power Normalization and $L_2$ Normalization to the fused features was essential for stable training and preventing output variance from element-wise multiplications.

Co-Attention and KLD Loss

To handle noisy features, the paper introduces a co-attention mechanism. It jointly learns to focus on relevant image regions and important question keywords simultaneously, ensuring visual features are conditioned on text and vice-versa.

For answer prediction, the model treats the task as a label distribution learning problem rather than simple classification. By using Kullback-Leibler Divergence (KLD) as a loss function, the model compares its predicted probability distribution against the distribution of answers provided by multiple human annotators. This allows the model to learn correlations between synonymous or similar answers, leading to faster convergence.

Evaluation and Results

The model was evaluated on the VQA-1.0 and VQA-2.0 benchmarks, using Visual Genome for data augmentation.

  • State-of-the-Art Performance: The MFH ensemble achieved 69.2% accuracy on the VQA-1.0 Test-Standard set, outperforming previous leaders like MCB (66.5%) and MLB (66.9%).
  • Efficiency: MFH achieved these results while using only 1/3 of the parameters and 2/3 of the GPU memory of the previous state-of-the-art (MCB).
  • VQA Challenge 2017: The model secured the runner-up spot in the 2017 VQA Challenge, with an accuracy of 68.16% on the VQA-2.0 test-challenge set.

Conclusion

MFH represents a significant step forward in multimodal fusion, demonstrating that high-order interactions can be captured efficiently. However, a gap remains between AI and human performance (~83.3%). While the model is excellent at pattern matching, it still struggles with multi-step logical reasoning and precise counting. Future work will focus on integrating external knowledge bases to assist with reasoning beyond the immediate visual pixels.

Summary updated

Abstract

Visual question answering (VQA) is challenging, because it requires a simultaneous understanding of both visual content of images and textual content of questions. To support the VQA task, we need to find good solutions for the following three issues: 1) fine-grained feature representations for both the image and the question; 2) multimodal feature fusion that is able to capture the complex interactions between multimodal features; and 3) automatic answer prediction that is able to consider the complex correlations between multiple diverse answers for the same question. For fine-grained image and question representations, a “coattention” mechanism is developed using a deep neural network (DNN) architecture to jointly learn the attentions for both the image and the question, which can allow us to reduce the irrelevant features effectively and obtain more discriminative features for image and question representations. For multimodal feature fusion, a generalized multimodal factorized high-order pooling approach (MFH) is developed to achieve more effective fusion of multimodal features by exploiting their correlations sufficiently, which can further result in superior VQA performance as compared with the state-of-the-art approaches. For answer prediction, the Kullback-Leibler divergence is used as the loss function to achieve precise characterization of the complex correlations between multiple diverse answers with the same or similar meaning, which can allow us to achieve faster convergence rate and obtain slightly better accuracy on answer prediction. A DNN architecture is designed to integrate all these aforementioned modules into a unified model for achieving superior VQA performance. With an ensemble of our MFH models, we achieve the state-of-the-art performance on the large-scale VQA data sets and win the runner-up in VQA Challenge 2017.
BibTeX
@article{yu2018beyond,
  author = {Zhou Yu and Jun Yu and Chenchao Xiang and Jianping Fan and Dacheng Tao},
  title = {{Beyond Bilinear: Generalized Multimodal Factorized High-Order Pooling for Visual Question Answering}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2018},
  doi = {10.1109/tnnls.2018.2817340},
}
Figure