Abstract
Researchers have introduced KVQA, the first large-scale dataset for "World Knowledge-Aware" Visual Question Answering. Unlike traditional VQA, which typically queries generic objects (e.g., "How many chairs?"), KVQA requires identifying specific named entities and performing multi-hop reasoning over a Knowledge Graph to answer complex questions about their lives, relationships, and history.
Introduction
Standard Visual Question Answering (VQA) has historically focused on two areas: content-based VQA (e.g., "What color is the car?") and commonsense VQA (e.g., "What is used for amplifying sound?"). However, humans often ask questions that require world knowledge about specific people or places—information that isn't contained within the image pixels themselves.
For example, looking at a photo of two politicians and asking, "Who is the founder of the political party the person on the left belongs to?" requires identifying the person, finding their party in a database, and then identifying that party's founder. This paper addresses the lack of datasets and models capable of this "multi-hop" reasoning involving named entities.
Methodology
The authors propose a two-step pipeline to tackle the KVQA challenge:
- Visual Entity Linking: Before answering a question, the system must identify the individuals in the image. The authors used FaceNet for face recognition and MS-Captionbot to extract names from image captions, testing this in both "Closed-world" (18K known people) and "Open-world" (69K potential people) settings.
- VQA over Knowledge Graphs (KG): Once entities are identified, the system queries Wikidata to retrieve relevant facts (up to 3 "hops" away). They implemented a Memory Network (MemNet) as a baseline, which stores these facts in a "memory" and uses an attention mechanism to decide which facts are relevant to the specific question.
The KVQA Dataset
The KVQA dataset is significantly larger (over 12x) than previous knowledge-based datasets like FVQA or KB-VQA. It contains 183,007 question-answer pairs across 24,602 images, featuring over 18,000 unique named entities. Images and captions were harvested from Wikipedia, while facts were pulled from Wikidata.
Key Insights and Results
- The Entity Bottleneck: The primary hurdle is "Visual Entity Linking." If the model cannot correctly identify the person in the photo, it cannot query the correct facts from the Knowledge Graph.
- Spatial and Multi-Hop Reasoning: The dataset forces models to combine spatial awareness ("person on the left") with external data, preventing models from "cheating" by guessing based on the most famous person in the frame.
- Performance Gap: When the model was told exactly who was in the image (the "Oracle" setting), accuracy reached 59.2%. In the more difficult "Open-world" setting, performance dropped to 36.8%.
- Model Comparison: Memory Networks significantly outperformed standard BLSTMs, proving that explicitly storing and attending to KG facts is essential for this task.
Conclusion and Future Work
While KVQA marks a major advancement, it is currently limited primarily to persons. Future work should expand to include monuments, events, and other named entity types. Additionally, scaling face identification to millions of people remains an unsolved challenge. The authors suggest exploring end-to-end systems where the question helps guide the visual identification process directly.
Summary updated