TL;DR
This paper introduces RG-VQA, a novel approach to Visual Question Answering (VQA) that leverages Dense Passage Retrievers (DPRs) to efficiently extract relevant knowledge from structured knowledge bases. By integrating a retriever-generator pipeline, RG-VQA significantly boosts the reasoning capabilities of VQA systems, achieving state-of-the-art performance on the challenging ScienceQA dataset and outperforming GPT-4.
What Problem Does This Paper Address?
Visual Question Answering (VQA) systems are designed to answer questions about images, but current systems struggle with complex, knowledge-intensive questions that require reasoning beyond what's directly visible. While integrating knowledge graphs (KGs) and chain-of-thought (CoT) reasoning has shown promise, these methods often face challenges with scalability due to the complexity of graph neural networks (GNNs) and can lead to high inference times. This paper addresses the need for a more efficient and scalable method to infuse external knowledge into VQA models for enhanced reasoning.
Methods
The core of RG-VQA is a retriever-generator pipeline. Instead of relying on GNNs for KG integration, RG-VQA repurposes Dense Passage Retrievers (DPRs), specifically ColBERTv2, to efficiently retrieve relevant information from structured knowledge bases. The process involves:
- Image Description Generation: A Vision-Language Model (VLM) like LLaVA generates descriptions for each image. These descriptions, along with the question and answer choices, form the query for the retriever.
- Weakly Supervised Retriever Training: ColBERTv2 is trained iteratively over multiple rounds using a "Relevance Guided Supervision" (RGS) heuristic. This heuristic identifies relevant triples from the KG by checking for significant overlap with nouns and adjectives in the question and correct answer. This approach aims to improve the retriever's ability to find knowledge pertinent to answering the question.
- Generator Training: The VLM is then trained using the image descriptions, the original question, answer choices, and the top N triples retrieved by the trained ColBERTv2 model as context. This allows the VLM to leverage the extracted knowledge for answering questions and generating rationales.
Datasets
The primary dataset used for evaluation is ScienceQA (Lu et al., 2022), a large-scale multimodal dataset for science question answering that requires external knowledge and complex reasoning. The authors also mention evaluating on A-OKVQA (Schwenk et al., 2022) to demonstrate generalizability to other knowledge-intensive VQA datasets.
Key Insights and Ideas
- DPRs as a Viable Alternative to GNNs for KG Reasoning: The paper demonstrates that DPRs, when trained appropriately, can effectively substitute GNNs for knowledge retrieval from structured KGs, offering a more scalable and computationally efficient solution.
- Iterative Retriever Training is Crucial: The study highlights that simply using an off-the-shelf retriever is insufficient. Iterative training of the retriever, guided by relevant knowledge, significantly improves VQA performance, leading to better alignment between retrieved triples and the question.
- Image Descriptions Enhance Retriever Training: Including image descriptions in the retriever's input significantly boosts performance, underscoring their importance in guiding the retrieval process for knowledge-intensive VQA.
- RG-VQA Scales Well: The proposed pipeline is shown to be effective across various VLM scales, from 250M to 13B parameters, and maintains reasonable inference latencies.
Key Results
- RG-VQA achieves a state-of-the-art test accuracy of 92.10% on the ScienceQA dataset, outperforming GPT-4 by over 8%.
- The method shows a consistent improvement of over 2% across various VLMs when the retriever is iteratively trained.
- Retriever training leads to an improvement of nearly 2.5% in zero-shot testing compared to using an off-the-shelf retriever.
- RG-VQA achieves comparable performance to GNN-based methods like KAM-CoT while using significantly fewer trainable parameters and offering much faster inference times (80 minutes for RG-VQA vs. 116 minutes for KAM-CoT on the ScienceQA test set).
- On the A-OKVQA dataset, RG-VQA significantly outperforms prior approaches, achieving an accuracy of 77.38%.
Limitations and Future Work
- Insufficient Knowledge Triples for Linguistic Questions: The current knowledge graph (ConceptNet) lacks sufficient triples to adequately address questions requiring deep linguistic understanding.
- Inability to Recognize Named Entities in Images: The pipeline cannot currently recognize named entities directly within images, which could be crucial for certain datasets.
- Future Directions:
- Employing well-trained vision encoders with multimodal Knowledge Graphs to retrieve relevant multimodal context about entities.
- Integrating a visual retriever to directly incorporate image features in the initial steps.
- Utilizing KGs with multimodal nodes, rather than just text-only nodes.
Summary updated