Abstract
This paper introduces Multimodal Factorized High-order pooling (MFH), a novel method for fusing visual and textual information in Visual Question Answering (VQA). By cascading multiple factorized bilinear pooling blocks, the authors capture complex, high-order interactions between modalities. When combined with a dual "co-attention" mechanism and a distribution-based loss function, MFH achieved state-of-the-art results and secured the runner-up spot in the 2017 VQA Challenge.
Introduction
Visual Question Answering (VQA) is a complex multimodal task requiring a computer to interpret an image and answer a natural language question. This task is significantly more challenging than image captioning as it requires fine-grained reasoning and the ability to distinguish relevant information from noise.
The authors identify three major bottlenecks in existing VQA systems:
- Noisy Features: Global image features often contain irrelevant clutter, while questions contain linguistic filler.
- Ineffective Fusion: Standard methods like concatenation or addition fail to capture nuanced, non-linear correlations between pixels and words.
- Rigid Answer Prediction: Treating unique answers as separate, unrelated classes ignores semantic similarities (e.g., "puppy" vs. "little dog"), making training less efficient.
Methodology
The proposed architecture addresses these bottlenecks through three key innovations integrated into a unified deep learning framework.
Multimodal Factorized High-order Pooling (MFH)
While previous state-of-the-art methods utilized "bilinear pooling" (outer products) to fuse modalities, the approach was computationally expensive. The authors first propose MFB (Multimodal Factorized Bilinear pooling), which uses matrix factorization to achieve bilinear expressiveness with fewer parameters. They then extend this to MFH by cascading multiple MFB blocks. This allows the model to learn "high-order" relationships—essentially capturing interactions between interactions.
Critical to this fusion is normalization; the authors found that applying Power Normalization and $L_2$ Normalization to the fused features was essential for stable training and preventing output variance from element-wise multiplications.
Co-Attention and KLD Loss
To handle noisy features, the paper introduces a co-attention mechanism. It jointly learns to focus on relevant image regions and important question keywords simultaneously, ensuring visual features are conditioned on text and vice-versa.
For answer prediction, the model treats the task as a label distribution learning problem rather than simple classification. By using Kullback-Leibler Divergence (KLD) as a loss function, the model compares its predicted probability distribution against the distribution of answers provided by multiple human annotators. This allows the model to learn correlations between synonymous or similar answers, leading to faster convergence.
Evaluation and Results
The model was evaluated on the VQA-1.0 and VQA-2.0 benchmarks, using Visual Genome for data augmentation.
- State-of-the-Art Performance: The MFH ensemble achieved 69.2% accuracy on the VQA-1.0 Test-Standard set, outperforming previous leaders like MCB (66.5%) and MLB (66.9%).
- Efficiency: MFH achieved these results while using only 1/3 of the parameters and 2/3 of the GPU memory of the previous state-of-the-art (MCB).
- VQA Challenge 2017: The model secured the runner-up spot in the 2017 VQA Challenge, with an accuracy of 68.16% on the VQA-2.0 test-challenge set.
Conclusion
MFH represents a significant step forward in multimodal fusion, demonstrating that high-order interactions can be captured efficiently. However, a gap remains between AI and human performance (~83.3%). While the model is excellent at pattern matching, it still struggles with multi-step logical reasoning and precise counting. Future work will focus on integrating external knowledge bases to assist with reasoning beyond the immediate visual pixels.
Summary updated