TL;DR
This paper tackles the challenge of Knowledge Base Visual Question Answering (KB-VQA) by proposing a supervised retrieval method for both external knowledge graphs and visual scene graphs. They demonstrate that explicitly retrieving and integrating this information significantly boosts performance for both traditional neural network models and large language models (LLMs), especially in multi-hop reasoning scenarios.
What Problem Does This Paper Address?
Visual Question Answering (VQA) aims to answer questions about an image. While impressive progress has been made, many VQA models struggle when the answer requires knowledge beyond what's directly visible in the image. This is where Knowledge Base Visual Question Answering (KB-VQA) comes in. In KB-VQA, models are given not only an image and a question but also access to an external knowledge base (like Wikipedia or ConceptNet) to find the necessary information.
The core problem this paper addresses is how to effectively leverage this external knowledge. Existing methods often rely on simple keyword matching for retrieval, which can be noisy and inefficient, leading to irrelevant information that hinders the reasoning process. Furthermore, while large language models (LLMs) possess a vast amount of implicit knowledge, their ability to integrate explicit external knowledge and perform complex multi-hop reasoning remains a challenge. The authors aim to bridge this gap by developing a more robust retrieval mechanism and analyzing how it impacts both specialized VQA architectures and LLM-based approaches.
Methods
The paper proposes a two-pronged approach:
- Supervised Knowledge Retrieval: They introduce a supervised retrieval model trained using contrastive loss. This model learns to maximize the similarity between a question's embedding and the embeddings of relevant knowledge triplets (from an external knowledge graph) or visual scene graph triplets. This is a departure from heuristic-based retrieval, aiming for semantically relevant information.
- Dual Frameworks for Integration and Reasoning:
- Task-Specific Neural Architecture (NN): They design a neural network that iteratively updates a query using cross-attention mechanisms to integrate retrieved knowledge graph and scene graph triplets. This allows for multi-hop reasoning by attending to both external and visual information.
- Large Language Model (LLM) Backbone: They leverage LLMs by incorporating retrieved knowledge triplets (converted into sentences) and image captions into the LLM's prompt. This allows them to analyze how LLMs perform with explicit external knowledge compared to their implicit knowledge.
Datasets
The primary dataset used in this work is KRVQA (Knowledge-Routed Visual Question Reasoning). This dataset is notable because it provides not only questions and answers but also "supporting reasons" that indicate which parts of the answer come from visual information and which come from external knowledge. This meta-data is crucial for training and evaluating their supervised retrieval modules. The authors also mention using the Visual Genome dataset for its scene graphs.
Key Insights and Ideas
- Supervised Retrieval is Key: Explicitly training a retrieval module using contrastive loss significantly outperforms unsupervised methods (like simple cosine similarity of text embeddings) for both knowledge graph and scene graph retrieval.
- LLMs Still Need External Knowledge: While LLMs excel at 1-hop reasoning and show effectiveness for KB-related questions due to their implicit knowledge, they struggle with 2-hop reasoning compared to fine-tuned neural networks, even when provided with explicit knowledge. This highlights that LLMs don't fully alleviate the need for external KBs.
- Visual Data Needs High-Level Representation: Using scene graphs (a high-level, object-centric representation) for visual information yields much better results than relying solely on low-level image features or captions. This underscores the importance of structured visual representations.
- Multi-hop Reasoning is a Bottleneck: Both NN and LLM models show weaknesses in complex multi-hop reasoning, especially when the "head entity" (the primary subject of the query) is missing in the retrieved facts.
Key Results
- The supervised retrieval module achieved a top-1 accuracy of 59.10% for KB retrieval and 60.00% for SG retrieval, a substantial improvement over unsupervised methods (17.5% and 27.12% respectively).
- The proposed NN model with both KB and SG retrieval achieved an accuracy of 44.36% on the KRVQA dataset, outperforming previous NN models like DMMGR (31.8%).
- The LLM model enriched with retrieved knowledge achieved 40.50% accuracy, a significant improvement over the base LLM (Promptcap at 19.8%).
- For 1-hop questions, LLMs generally outperformed the NN models. However, for 2-hop questions, the NN model showed superior performance.
Limitations and Future Work
- Static Retrieval: The current retrieval is performed once at the beginning. Future work will explore dynamic retrieval that refines knowledge after each reasoning step.
- Scene Graph Approximation: The study used human-annotated scene graphs. Future work will investigate using approximated scene graphs, acknowledging that current approximation methods lead to significant accuracy drops.
- Dataset Limitations: Datasets like OK-VQA lack supporting facts/reasons, making detailed analysis difficult. The authors plan to adapt their models for such datasets.
- Interchained Retrieval and Reasoning: The retrieval and reasoning parts were executed separately. Future work will investigate interchaining these processes for more dynamic interaction.
Summary updated