KVQA: Knowledge-Aware Visual Question Answering

Sanket Shah, Anand Mishra, Naganand Yadati, Partha Pratim TalukdarAAAI 2019 · DOI

Abstract

Researchers have introduced KVQA, the first large-scale dataset for "World Knowledge-Aware" Visual Question Answering. Unlike traditional VQA, which typically queries generic objects (e.g., "How many chairs?"), KVQA requires identifying specific named entities and performing multi-hop reasoning over a Knowledge Graph to answer complex questions about their lives, relationships, and history.

Introduction

Standard Visual Question Answering (VQA) has historically focused on two areas: content-based VQA (e.g., "What color is the car?") and commonsense VQA (e.g., "What is used for amplifying sound?"). However, humans often ask questions that require world knowledge about specific people or places—information that isn't contained within the image pixels themselves.

For example, looking at a photo of two politicians and asking, "Who is the founder of the political party the person on the left belongs to?" requires identifying the person, finding their party in a database, and then identifying that party's founder. This paper addresses the lack of datasets and models capable of this "multi-hop" reasoning involving named entities.

The KVQA framework links visual entities to a knowledge graph for multi-hop reasoning.

Methodology

The authors propose a two-step pipeline to tackle the KVQA challenge:

  1. Visual Entity Linking: Before answering a question, the system must identify the individuals in the image. The authors used FaceNet for face recognition and MS-Captionbot to extract names from image captions, testing this in both "Closed-world" (18K known people) and "Open-world" (69K potential people) settings.
  2. VQA over Knowledge Graphs (KG): Once entities are identified, the system queries Wikidata to retrieve relevant facts (up to 3 "hops" away). They implemented a Memory Network (MemNet) as a baseline, which stores these facts in a "memory" and uses an attention mechanism to decide which facts are relevant to the specific question.

The KVQA Dataset

The KVQA dataset is significantly larger (over 12x) than previous knowledge-based datasets like FVQA or KB-VQA. It contains 183,007 question-answer pairs across 24,602 images, featuring over 18,000 unique named entities. Images and captions were harvested from Wikipedia, while facts were pulled from Wikidata.

Distribution of question types in the KVQA dataset, highlighting the diversity of queries.

Key Insights and Results

  • The Entity Bottleneck: The primary hurdle is "Visual Entity Linking." If the model cannot correctly identify the person in the photo, it cannot query the correct facts from the Knowledge Graph.
  • Spatial and Multi-Hop Reasoning: The dataset forces models to combine spatial awareness ("person on the left") with external data, preventing models from "cheating" by guessing based on the most famous person in the frame.
  • Performance Gap: When the model was told exactly who was in the image (the "Oracle" setting), accuracy reached 59.2%. In the more difficult "Open-world" setting, performance dropped to 36.8%.
  • Model Comparison: Memory Networks significantly outperformed standard BLSTMs, proving that explicitly storing and attending to KG facts is essential for this task.

Conclusion and Future Work

While KVQA marks a major advancement, it is currently limited primarily to persons. Future work should expand to include monuments, events, and other named entity types. Additionally, scaling face identification to millions of people remains an unsolved challenge. The authors suggest exploring end-to-end systems where the question helps guide the visual identification process directly.

Summary updated

BibTeX
@inproceedings{shah2019kvqa,
  author = {Sanket Shah and Anand Mishra and Naganand Yadati and Partha Pratim Talukdar},
  title = {{KVQA: Knowledge-Aware Visual Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33018876},
}
Figure