Abstract
Visual Question Answering (VQA) is a multimodal task involving Computer Vision (CV) and Natural Language Processing (NLP), the goal is to establish a high-efficiency VQA model. Learning a fine-grained and simultaneous understanding of both the visual content of images and the textual content of questions is the heart of VQA. In this paper, a novel Multimodal Encoder-Decoder Attention Networks (MEDAN) is proposed. The MEDAN consists of Multimodal Encoder-Decoder Attention (MEDA) layers cascaded in depth, and can capture rich and reasonable question features and image features by associating keywords in question with important object regions in image. Each MEDA layer contains an Encoder module modeling the self-attention of questions, as well as a Decoder module modeling the question-guided-attention and self-attention of images. Experimental evaluation results on the benchmark VQA-v2 dataset demonstrate that MEDAN achieves state-of-the-art VQA performance. With the Adam solver, our best single model delivers 71.01% overall accuracy on the test-std set, and with the AdamW solver, we achieve an overall accuracy of 70.76% on the test-dev set. Additionally, extensive ablation studies are conducted to explore the reasons for MEDAN's effectiveness.
Introduction
Visual Question Answering (VQA) represents a significant milestone in artificial intelligence, requiring a model to bridge the gap between Computer Vision and Natural Language Processing. To answer a question about an image correctly, a model must not only identify objects but also understand their spatial relationships and attributes as dictated by the textual query.
The core challenge in VQA is co-attention: the ability of a model to dynamically focus on relevant image regions based on specific words in the question. While early models utilized coarse attention mechanisms, they often failed to capture the dense, word-by-region interactions necessary for complex reasoning. Newer "dense" co-attention models have improved performance but often struggle to maintain gains when scaled in depth. This paper introduces the Multimodal Encoder-Decoder Attention Network (MEDAN), which addresses these limitations through a cascaded architecture designed to mimic human reasoning patterns.
Methodology: The MEDAN Architecture
The primary innovation of this work is the Multimodal Encoder-Decoder Attention (MEDA) layer. These layers are cascaded $N$ times to allow for iterative refinement of features.
The MEDA Layer
Each MEDA layer is composed of two main modules: an Encoder and a Decoder.
- The Encoder: This module processes the textual input. Using multi-head self-attention ($SA_E$), it models the relationships between words in the question to identify key linguistic features.
- The Decoder: The decoder is the heart of the multimodal interaction. It performs reasoning in two distinct steps:
- Question-Guided Attention ($GA_{EV}$): The refined question features from the encoder are used to query the image, identifying relevant object regions.
- Image Self-Attention ($SA_V$): Once relevant regions are identified, this module models the relationships between those specific regions (e.g., the interaction between a "person" and a "frisbee").
Cascading and Feature Extraction
By stacking these layers (with $N=6$ found to be optimal), MEDAN performs iterative reasoning. Image features are extracted using a Faster R-CNN pre-trained on the Visual Genome dataset, while question features are initialized with GloVe embeddings and processed through an LSTM.
Experimental Results
The MEDAN model was evaluated on the benchmark VQA-v2 dataset, which contains over 443,000 question-answer pairs.
Performance Benchmarks
MEDAN achieved state-of-the-art performance, reaching 71.01% overall accuracy on the VQA-v2 test-std set using the Adam solver. Interestingly, while AdamW is often preferred for its generalization properties, the standard Adam solver yielded the highest single-model accuracy in this specific configuration (70.76% on test-dev with AdamW vs higher with Adam).
Ablation Studies
Extensive ablation studies confirmed the effectiveness of the cascaded design. Increasing the number of layers from 1 to 6 improved validation accuracy from 65.49% to 67.08%, proving that the model benefits significantly from depth. Additionally, testing various numbers of attention heads ($h$) showed that 8 heads provided the optimal balance between computational efficiency and accuracy.
Conclusion
The Multimodal Encoder-Decoder Attention Network (MEDAN) demonstrates that the order of reasoning—filtering image content via question-guided attention before modeling image relationships—is crucial for high-performance VQA. By leveraging a deep, cascaded architecture, MEDAN achieves state-of-the-art results on the VQA-v2 benchmark. Future work may explore combining this encoder-decoder structure with gated self-attention mechanisms to further enhance keyword sensitivity and spatial reasoning.
Summary updated