Textually Enriched Neural Module Networks for Visual Question Answering

Khyathi Raghavi Chandu, Mary Arpita Pyreddy, Matthieu Felix, Narendra Nath JoshiarXiv 2018 · OpenReview

Abstract

Problems at the intersection of language and vision, like visual question answering, have recently been gaining a lot of attention in the field of multi-modal machine learning as computer vision research moves beyond traditional recognition tasks. There has been recent success in visual question answering using deep neural network models which use the linguistic structure of the questions to dynamically instantiate network layouts. In the process of converting the question to a network layout, the question is simplified, which results in loss of information in the model. In this paper, we enrich the image information with textual data using image captions and external knowledge bases to generate more coherent answers. We achieve 57.1% overall accuracy on the test-dev open-ended questions from the visual question answering (VQA 1.0) real image dataset.

Introduction

Visual Question Answering (VQA) is a complex multi-modal task requiring a system to provide natural language answers to questions about visual content. A leading approach involves Neural Module Networks (NMNs), which dynamically assemble a neural network layout based on the linguistic structure of the query. However, NMNs often struggle with information loss during the simplification of questions into logical parses and lack the "common sense" or external knowledge required for many real-world questions.

This paper explores methods to enrich NMNs with extra textual context—specifically image captions and external knowledge from DBPedia—to mitigate these weaknesses and improve answer coherence.

Methods

The authors propose three distinct strategies for textual enrichment within the NMN framework:

1. Caption Information

A pre-trained image captioning module generates a textual description of the image. This caption is encoded via an LSTM into a context vector, which is then fused with the NMN's output and the question's context vector before the final answer prediction. This serves as a semantic bridge, providing descriptive details that might be lost in the NMN's dependency parse.

Architecture of the NMN with Caption Information enrichment.

2. Caption Attention

Recognizing that only specific parts of a caption may be relevant to a given question, the authors implemented an attention mechanism. This allows the model to selectively focus on words in the caption that align with the question's intent, rather than treating the caption as a monolithic block of text.

Architecture of the NMN with Caption Attention mechanism.

3. External Knowledge Sources

For questions requiring general knowledge, the system queries a database of DBPedia abstracts. Nouns extracted from the question are used to retrieve relevant abstracts via Apache Lucene. These abstracts are converted into vectors using Doc2Vec and used to seed the LSTM that parses the question, providing the model with necessary outside context.

Key Results

The enriched models were evaluated against a baseline NMN on the VQA 1.0 real-image dataset. The results indicate that simple caption integration provides the most significant performance boost.

Model Accuracy (%)
Baseline NMN 56.9
NMN + Caption Information 57.1
NMN + Caption Attention 56.6
NMN + External Knowledge 56.8

Qualitative analysis revealed that captioning data was particularly instrumental in helping the model identify specific objects (such as "two planes" or a "toothbrush") that the base NMN failed to detect visually.

Discussion and Conclusion

A key insight from this work is the trade-off between information and noise. Surprisingly, the "Caption Attention" model performed slightly worse than the simpler "Caption Information" model, likely because the attention mechanism introduced noise when the generated captions were not perfectly relevant to the question. Furthermore, experiments with "Longest Parses" (retaining more question structure) actually decreased performance, confirming that current NMNs struggle with complex layouts.

Future work should focus on question-guided captioning—generating descriptions tailored to the specific information requested by the user—to ensure the textual enrichment remains relevant and high-quality.

Summary updated

Abstract

Problems at the intersection of language and vision, like visual question answering, have recently been gaining a lot of attention in the field of multi-modal machine learning as computer vision research moves beyond traditional recognition tasks. There has been recent success in visual question answering using deep neural network models which use the linguistic structure of the questions to dynamically instantiate network layouts. In the process of converting the question to a network layout, the question is simplified, which results in loss of information in the model. In this paper, we enrich the image information with textual data using image captions and external knowledge bases to generate more coherent answers. We achieve 57.1% overall accuracy on the test-dev open-ended questions from the visual question answering (VQA 1.0) real image dataset.
BibTeX
@misc{chandu2018textually,
  author = {Khyathi Raghavi Chandu and Mary Arpita Pyreddy and Matthieu Felix and Narendra Nath Joshi},
  title = {{Textually Enriched Neural Module Networks for Visual Question Answering}},
  year = {2018},
  eprint = {1809.08697},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.08697},
}
Figure