Abstract
Problems at the intersection of language and vision, like visual question answering, have recently been gaining a lot of attention in the field of multi-modal machine learning as computer vision research moves beyond traditional recognition tasks. There has been recent success in visual question answering using deep neural network models which use the linguistic structure of the questions to dynamically instantiate network layouts. In the process of converting the question to a network layout, the question is simplified, which results in loss of information in the model. In this paper, we enrich the image information with textual data using image captions and external knowledge bases to generate more coherent answers. We achieve 57.1% overall accuracy on the test-dev open-ended questions from the visual question answering (VQA 1.0) real image dataset.
Introduction
Visual Question Answering (VQA) is a complex multi-modal task requiring a system to provide natural language answers to questions about visual content. A leading approach involves Neural Module Networks (NMNs), which dynamically assemble a neural network layout based on the linguistic structure of the query. However, NMNs often struggle with information loss during the simplification of questions into logical parses and lack the "common sense" or external knowledge required for many real-world questions.
This paper explores methods to enrich NMNs with extra textual context—specifically image captions and external knowledge from DBPedia—to mitigate these weaknesses and improve answer coherence.
Methods
The authors propose three distinct strategies for textual enrichment within the NMN framework:
1. Caption Information
A pre-trained image captioning module generates a textual description of the image. This caption is encoded via an LSTM into a context vector, which is then fused with the NMN's output and the question's context vector before the final answer prediction. This serves as a semantic bridge, providing descriptive details that might be lost in the NMN's dependency parse.
2. Caption Attention
Recognizing that only specific parts of a caption may be relevant to a given question, the authors implemented an attention mechanism. This allows the model to selectively focus on words in the caption that align with the question's intent, rather than treating the caption as a monolithic block of text.
3. External Knowledge Sources
For questions requiring general knowledge, the system queries a database of DBPedia abstracts. Nouns extracted from the question are used to retrieve relevant abstracts via Apache Lucene. These abstracts are converted into vectors using Doc2Vec and used to seed the LSTM that parses the question, providing the model with necessary outside context.
Key Results
The enriched models were evaluated against a baseline NMN on the VQA 1.0 real-image dataset. The results indicate that simple caption integration provides the most significant performance boost.
| Model | Accuracy (%) |
|---|---|
| Baseline NMN | 56.9 |
| NMN + Caption Information | 57.1 |
| NMN + Caption Attention | 56.6 |
| NMN + External Knowledge | 56.8 |
Qualitative analysis revealed that captioning data was particularly instrumental in helping the model identify specific objects (such as "two planes" or a "toothbrush") that the base NMN failed to detect visually.
Discussion and Conclusion
A key insight from this work is the trade-off between information and noise. Surprisingly, the "Caption Attention" model performed slightly worse than the simpler "Caption Information" model, likely because the attention mechanism introduced noise when the generated captions were not perfectly relevant to the question. Furthermore, experiments with "Longest Parses" (retaining more question structure) actually decreased performance, confirming that current NMNs struggle with complex layouts.
Future work should focus on question-guided captioning—generating descriptions tailored to the specific information requested by the user—to ensure the textual enrichment remains relevant and high-quality.
Summary updated