OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge.

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, Roozbeh Mottaghi

Abstract

Visual Question Answering (VQA) in its ideal form lets us study reasoning in the joint space of vision and language and serves as a proxy for the AI task of scene understanding. However, most VQA benchmarks to date are focused on questions such as simple counting, visual attributes, and object detection that do not require reasoning or knowledge beyond what is in the image. In this paper, we address the task of knowledge-based visual question answering and provide a benchmark, called OK-VQA, where the image content is not sufficient to answer the questions, encouraging methods that rely on external knowledge resources. Our new dataset includes more than 14,000 questions that require external knowledge to answer. We show that the performance of the state-of-the-art VQA models degrades drastically in this new setting. Our analysis shows that our knowledge-based VQA task is diverse, difficult, and large compared to previous knowledge-based VQA datasets. We hope that this dataset enables researchers to open up new avenues for research in this domain.

Introduction

Visual Question Answering (VQA) has traditionally focused on tasks like counting objects or identifying colors—tasks that can be solved by analyzing image pixels alone. However, true scene understanding requires more than just "eyesight"; it requires world knowledge. For instance, while a model can identify a "teddy bear," it requires external historical knowledge to know the toy is named after Theodore Roosevelt.

Figure 1: Teddy bears and external knowledge.

The OK-VQA (Outside Knowledge Visual Question Answering) paper introduces a benchmark to address this gap, forcing models to consult external, unstructured knowledge sources to answer complex questions about images.

The OK-VQA Dataset

The primary contribution is a dataset of 14,055 questions across 14,031 images from the COCO dataset. Unlike previous benchmarks that used structured "fact triplets," OK-VQA is designed for the "open world," where answers might be found in unstructured text like Wikipedia.

Figure 2: OK-VQA categories and examples.

The dataset covers 10 diverse domains, including History, Science & Technology, and Cooking. Questions were curated via crowdsourcing to ensure they require reasoning beyond the visual content.

Methodology: ArticleNet

To establish a baseline, the authors developed ArticleNet (AN). This system generates search queries from the image and question, retrieves relevant Wikipedia articles, and uses a GRU-based architecture to extract the most likely answer. This retrieval system was integrated with state-of-the-art VQA models like MUTAN and BAN to evaluate the impact of external information on visual reasoning.

Key Results and Insights

  • The "Knowledge Gap": Analysis showed that 78% of questions in standard datasets like VQA v2 are answerable by a child, highlighting the need for more cognitively demanding benchmarks.
  • Performance Drop: State-of-the-art models scoring ~70% on standard benchmarks plummeted to ~25-26% on OK-VQA, revealing a significant bottleneck in reasoning.
  • Retrieval Boost: Adding ArticleNet to the MUTAN model provided a small but significant accuracy boost (from 26.41% to 27.84%), proving that external text provides complementary data.
  • Vision vs. Reasoning: Improving visual features (e.g., upgrading to ResNet152) yielded only marginal gains, suggesting that the primary challenge in OK-VQA is knowledge retrieval and reasoning rather than perception.

Conclusion

OK-VQA sets a new, challenging standard for the VQA community. While retrieval systems like ArticleNet show promise in domains like "Brands" or "Science," there remains a massive gap between AI and human performance, particularly in common-sense reasoning. Future research must look beyond Wikipedia to incorporate broader world knowledge and more sophisticated natural language understanding.

Summary updated

Abstract

Visual Question Answering (VQA) in its ideal form lets us study reasoning in the joint space of vision and language and serves as a proxy for the AI task of scene understanding. However, most VQA benchmarks to date are focused on questions such as simple counting, visual attributes, and object detection that do not require reasoning or knowledge beyond what is in the image. In this paper, we address the task of knowledge-based visual question answering and provide a benchmark, called OK-VQA, where the image content is not sufficient to answer the questions, encouraging methods that rely on external knowledge resources. Our new dataset includes more than 14,000 questions that require external knowledge to answer. We show that the performance of the state-of-the-art VQA models degrades drastically in this new setting. Our analysis shows that our knowledge-based VQA task is diverse, difficult, and large compared to previous knowledge-based VQA datasets. We hope that this dataset enables researchers to open up new avenues for research in this domain.
BibTeX
@inproceedings{marino2019ok,
  author = {Kenneth Marino and Mohammad Rastegari and Ali Farhadi and Roozbeh Mottaghi},
  title = {{OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge.}},
  booktitle = {CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=HobDHXXxdpS},
}
Figure