Beyond Bilinear: Generalized Multi-modal Factorized High-order Pooling for Visual Question Answering

Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, Dacheng Tao

Abstract

Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both visual content of images and textual content of questions. To support the VQA task, we need to find good solutions for the following three issues: 1) fine-grained feature representations for both the image and the question; 2) multi-modal feature fusion that is able to capture the complex interactions between multi-modal features; 3) automatic answer prediction that is able to consider the complex correlations between multiple diverse answers for the same question. This paper introduces a generalized Multi-modal Factorized High-order pooling approach (MFH) to achieve more effective fusion of multi-modal features. Combined with a co-attention mechanism and KL-divergence loss, our unified deep neural network architecture achieves state-of-the-art performance on large-scale VQA datasets and secured the runner-up position in the VQA Challenge 2017.

Introduction: The VQA Challenge

Visual Question Answering (VQA) requires a machine to understand an image and a natural language question simultaneously to produce a correct answer. Despite significant progress, three critical bottlenecks persist in existing VQA systems:

  1. Feature Representation: Standard global features often include "noise" from irrelevant parts of an image or question.
  2. Multimodal Fusion: Combining image and text features is usually done via simple concatenation or element-wise addition, which fails to capture complex correlations. While bilinear pooling methods like Multimodal Compact Bilinear (MCB) pooling were an improvement, they are often computationally expensive.
  3. Answer Correlation: Most models treat answers as independent classes, ignoring semantic overlaps (e.g., "puppy" and "dog").
The general VQA framework involves feature representation, multi-modal fusion, and answer prediction.

Methodology

The proposed architecture integrates three key modules into a unified model to address the aforementioned challenges.

Multimodal Factorized High-order Pooling (MFH)

Building on Multimodal Factorized Bilinear (MFB) pooling, we developed MFH. MFB factorizes a massive projection matrix into two low-rank matrices, creating an efficient "Expand and Squeeze" process. MFH extends this by cascading multiple MFB blocks, allowing the model to learn higher-order interactions between features that simple bilinear models miss.

The MFH architecture cascades multiple MFB blocks to capture high-order feature interactions.

Co-Attention Mechanism

Instead of unidirectional attention, the model jointly learns to attend to both the image and the question. This "self-attentive" question module identifies key words to better guide the visual attention, effectively reducing noise in the feature representations.

KL-Divergence (KLD) Loss

Rather than traditional cross-entropy loss, we use KL-divergence to treat answer prediction as a label distribution learning problem. This approach penalizes the difference between the predicted probability distribution and the ground-truth distribution provided by multiple human annotators, accounting for the complex correlations between diverse answers.

Datasets and Evaluation

The model was evaluated on two primary benchmarks:

  • VQA-1.0: Approximately 200,000 images from MS-COCO with three questions per image.
  • VQA-2.0: A more balanced version designed to reduce language bias, used for the VQA Challenge 2017.

Key Insights and Results

  • High-Order Benefits: The transition from MFB (bilinear) to MFH (high-order) provided a consistent accuracy boost. Cascading two MFB blocks ($p=2$) was found to be the optimal configuration.
  • Normalization: Power Normalization and $L_2$ normalization are essential for stabilizing the output neurons in bilinear models, preventing wild fluctuations.
  • Performance and Efficiency: An ensemble of MFH models achieved 68.02% accuracy on the VQA-2.0 test-dev set. Notably, MFH outperformed previous state-of-the-art methods like MCB while using significantly fewer parameters and less GPU memory.
  • Ablation Success: On VQA-1.0, the MFH model achieved 66.8% accuracy, outperforming standard bilinear models like MLB (65.1%) and MCB (64.7%).

Conclusion

This work demonstrates that generalized high-order pooling is a powerful tool for multimodal feature fusion. By combining MFH with co-attention and distribution-based loss functions, VQA systems can better capture the nuances of both visual and textual data. Future work will explore integrating external knowledge bases to handle questions requiring common-sense reasoning beyond visual recognition.

Summary updated

Abstract

Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both visual content of images and textual content of questions. To support the VQA task, we need to find good solutions for the following three issues: 1) fine-grained feature representations for both the image and the question; 2) multi-modal feature fusion that is able to capture the complex interactions between multi-modal features; 3) automatic answer prediction that is able to consider the complex correlations between multiple diverse answers for the same question. For fine-grained image and question representations, a `co-attention' mechanism is developed by using a deep neural network architecture to jointly learn the attentions for both the image and the question, which can allow us to reduce the irrelevant features effectively and obtain more discriminative features for image and question representations. For multi-modal feature fusion, a generalized Multi-modal Factorized High-order pooling approach (MFH) is developed to achieve more effective fusion of multi-modal features by exploiting their correlations sufficiently, which can further result in superior VQA performance as compared with the state-of-the-art approaches. For answer prediction, the KL (Kullback-Leibler) divergence is used as the loss function to achieve precise characterization of the complex correlations between multiple diverse answers with the same or similar meaning, which can allow us to achieve faster convergence rate and obtain slightly better accuracy on answer prediction. A deep neural network architecture is designed to integrate all these aforementioned modules into a unified model for achieving superior VQA performance. With an ensemble of our MFH models, we achieve the state-of-the-art performance on the large-scale VQA datasets and win the runner-up in VQA Challenge 2017.
BibTeX
@article{yu2018beyond,
  author = {Zhou Yu and Jun Yu and Chenchao Xiang and Jianping Fan and Dacheng Tao},
  title = {{Beyond Bilinear: Generalized Multi-modal Factorized High-order Pooling for Visual Question Answering}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2018},
  doi = {10.1109/tnnls.2018.2817340},
}
Figure