Abstract
Visual question answering (VQA) is challenging because it requires a simultaneous understanding of both visual content of images and textual content of questions. To support the VQA task, we need to find good solutions for the following three issues: 1) fine-grained feature representations for both the image and the question; 2) multi-modal feature fusion that is able to capture the complex interactions between multi-modal features; 3) automatic answer prediction that is able to consider the complex correlations between multiple diverse answers for the same question. This paper introduces a generalized Multi-modal Factorized High-order pooling approach (MFH) to achieve more effective fusion of multi-modal features. Combined with a co-attention mechanism and KL-divergence loss, our unified deep neural network architecture achieves state-of-the-art performance on large-scale VQA datasets and secured the runner-up position in the VQA Challenge 2017.
Introduction: The VQA Challenge
Visual Question Answering (VQA) requires a machine to understand an image and a natural language question simultaneously to produce a correct answer. Despite significant progress, three critical bottlenecks persist in existing VQA systems:
- Feature Representation: Standard global features often include "noise" from irrelevant parts of an image or question.
- Multimodal Fusion: Combining image and text features is usually done via simple concatenation or element-wise addition, which fails to capture complex correlations. While bilinear pooling methods like Multimodal Compact Bilinear (MCB) pooling were an improvement, they are often computationally expensive.
- Answer Correlation: Most models treat answers as independent classes, ignoring semantic overlaps (e.g., "puppy" and "dog").
Methodology
The proposed architecture integrates three key modules into a unified model to address the aforementioned challenges.
Multimodal Factorized High-order Pooling (MFH)
Building on Multimodal Factorized Bilinear (MFB) pooling, we developed MFH. MFB factorizes a massive projection matrix into two low-rank matrices, creating an efficient "Expand and Squeeze" process. MFH extends this by cascading multiple MFB blocks, allowing the model to learn higher-order interactions between features that simple bilinear models miss.
Co-Attention Mechanism
Instead of unidirectional attention, the model jointly learns to attend to both the image and the question. This "self-attentive" question module identifies key words to better guide the visual attention, effectively reducing noise in the feature representations.
KL-Divergence (KLD) Loss
Rather than traditional cross-entropy loss, we use KL-divergence to treat answer prediction as a label distribution learning problem. This approach penalizes the difference between the predicted probability distribution and the ground-truth distribution provided by multiple human annotators, accounting for the complex correlations between diverse answers.
Datasets and Evaluation
The model was evaluated on two primary benchmarks:
- VQA-1.0: Approximately 200,000 images from MS-COCO with three questions per image.
- VQA-2.0: A more balanced version designed to reduce language bias, used for the VQA Challenge 2017.
Key Insights and Results
- High-Order Benefits: The transition from MFB (bilinear) to MFH (high-order) provided a consistent accuracy boost. Cascading two MFB blocks ($p=2$) was found to be the optimal configuration.
- Normalization: Power Normalization and $L_2$ normalization are essential for stabilizing the output neurons in bilinear models, preventing wild fluctuations.
- Performance and Efficiency: An ensemble of MFH models achieved 68.02% accuracy on the VQA-2.0 test-dev set. Notably, MFH outperformed previous state-of-the-art methods like MCB while using significantly fewer parameters and less GPU memory.
- Ablation Success: On VQA-1.0, the MFH model achieved 66.8% accuracy, outperforming standard bilinear models like MLB (65.1%) and MCB (64.7%).
Conclusion
This work demonstrates that generalized high-order pooling is a powerful tool for multimodal feature fusion. By combining MFH with co-attention and distribution-based loss functions, VQA systems can better capture the nuances of both visual and textual data. Future work will explore integrating external knowledge bases to handle questions requiring common-sense reasoning beyond visual recognition.
Summary updated