TL;DR
This paper introduces MSR2, a new benchmark for visual question answering (VQA) that specifically targets multi-entity reasoning and knowledge retrieval. They found that even state-of-the-art large vision-language models (LVLMs) struggle with these complex tasks, highlighting the need for better models that can integrate information from multiple sources and understand spatial relationships between multiple objects.
What Problem Does This Paper Address?
The field of Visual Question Answering (VQA) aims to enable AI models to answer questions about images. A more advanced form, Knowledge-Based VQA (KB-VQA), requires models to not only understand the image but also retrieve and integrate external knowledge to answer questions. While previous KB-VQA datasets have focused on commonsense knowledge or single entities, the authors argue that current large vision-language models (LVLMs) have become adept at these simpler tasks. They identify a gap in evaluating LVLMs' ability to handle more complex, real-world scenarios involving multiple entities within an image that require spatial reasoning and retrieval of non-commonsense knowledge from diverse sources. Existing benchmarks lack this multi-entity, knowledge-intensive focus, limiting our understanding of LVLM capabilities in these crucial areas.
Methods
The authors propose a novel dataset, MSR2, constructed through a multi-stage pipeline:
- Image Source: They start with the Openv7 dataset, filtering for images containing multiple objects of the same coarse category (e.g., multiple cars, multiple dogs).
- Entity Finding: GPT-4V is used to generate fine-grained object labels for entities within the selected images.
- Knowledge Retrieval: For each identified entity, relevant information is retrieved using BM25 and then filtered for relevance by GPT-4.
- Question Generation: GPT-4 generates questions and answers based on the entity labels and retrieved knowledge.
- Visual Question Generation: GPT-4V generates image captions, and GPT-4 rewrites the generated questions to incorporate visual context and map entities to image elements.
- Filtering: Both LLM-based and human evaluations (via Amazon MTurk) are employed to ensure the quality and naturalness of the generated data.
Datasets
- Training/Evaluation: The paper introduces a new benchmark dataset called MSR2.
- Existing Datasets (for comparison): FVQA, OKVQA, S3VQA, A-OKVQA, Encyclopedic VQA, InfoSeek.
MSR2 distinguishes itself by focusing on:
- Fine-grained Entities: Requiring models to recognize specific entities rather than broad categories.
- Knowledge Retrieval: Necessitating external knowledge beyond what's in the image.
- Multiple Entities: Questions involve interactions and relationships between several objects in the image.
Key Insights and Ideas
- LVLMs Struggle with Multi-Entity Spatial Reasoning: Even the most advanced LVLMs show significant limitations when asked questions that require understanding the spatial relationships between multiple objects in an image.
- External Knowledge is Crucial, but Not Sufficient: While integrating external knowledge boosts performance, it doesn't fully overcome the challenges of multi-entity reasoning.
- Fine-grained Entity Recognition is Key: Accurate identification of individual entities is a prerequisite for successful multi-entity reasoning.
- Mapping Entities to Questions is a Bottleneck: Models can possess relevant knowledge but fail to correctly link entities mentioned in the question to their visual representations.
Key Results
- Low Performance on MSR2: Existing LVLMs achieved only a ~10% improvement over a random baseline on MSR2. Pipeline methods (entity identification + LVLM) performed even worse.
- Oracle Baselines Show Potential: "Oracle" baselines, which provide entities and their mapping to the question, achieved significantly higher accuracy (up to 81.44% with wiki knowledge), demonstrating the potential of well-structured input.
- Spatial Reasoning Gap: A ~12.1% performance gap was observed between providing entities without mapping ('Oracle ent.') and providing entities with mapping ('Oracle'), highlighting the difficulty LVLMs face in spatial mapping.
- Knowledge Boost: Integrating external knowledge (wiki) improved performance by 4.47% over using only model parameters.
Limitations and Future Work
- Language Limitation: MSR2 is currently limited to English. Future work could extend it to multilingual settings.
- Dataset Diversity: The image sources lack diversity, particularly in containing multiple instances of the same broad category. Future work should explore more varied datasets to improve robustness and generalizability.
- Ambiguity and Partial Visibility: Some images may have partially occluded or out-of-frame entities, leading to ambiguous tagging and hindering fine-grained classification.
- Future Research: The authors anticipate MSR2 will inspire research into more generalized retrieval-augmented LVLMs, focusing on improving multi-entity spatial reasoning and knowledge integration.
Summary updated