OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, Roozbeh Mottaghi

Abstract

Visual Question Answering (VQA) serves as a critical proxy for scene understanding in AI. However, existing benchmarks often focus on simple visual recognition tasks like counting or attribute detection. This paper introduces OK-VQA (Outside Knowledge Visual Question Answering), a benchmark containing over 14,000 questions that require external knowledge to answer. By shifting the focus from "visual lookup" to "open-world reasoning," OK-VQA reveals a significant performance gap in state-of-the-art models, which struggle to link visual content with unstructured knowledge sources like Wikipedia.

The Problem: The Limits of Visual Lookup

Standard VQA models have become adept at parsing pixels to answer questions like "What color is the car?" or "How many apples are in the bowl?" Yet, true scene understanding requires more than just vision; it requires external knowledge.

A collection of stuffed animals. Answering questions about the historical origin of a "teddy bear" requires knowledge beyond the image pixels.

Consider an image of a teddy bear. A standard VQA model might identify the object, but asking "Which American president is associated with this stuffed animal?" requires the model to retrieve the historical fact that the toy was named after Theodore Roosevelt. Existing datasets often lack this requirement or rely on overly simplistic, structured knowledge triplets that fail to capture real-world complexity.

The OK-VQA Dataset

The researchers curated a dataset of 14,055 questions paired with 14,031 images from the COCO dataset. To ensure the benchmark truly tests external knowledge, the authors employed a multi-stage filtering process with human annotators to exclude questions that could be answered by visual inspection alone.

The questions are categorized into 10 diverse domains, including Science & Technology, History, and Weather & Climate.

Pedestrians using umbrellas during a snowfall. Questions in the "Weather & Climate" category might ask about the atmospheric conditions required for such precipitation.

Unlike previous efforts like FVQA, which use fixed knowledge graphs, OK-VQA encourages the use of unstructured, open-knowledge sources, forcing models to navigate the "messiness" of human information.

Methodology: Bridging Vision and Knowledge

To establish baselines, the authors tested state-of-the-art models like MUTAN and BAN. They also introduced ArticleNet (AN), a retrieval engine designed to generate search queries from questions and visual classifiers, retrieve relevant Wikipedia articles, and score sentences to identify the most likely answer. Hybrid models (e.g., MUTAN + AN) were then created to feed this retrieved knowledge back into the visual reasoning engine.

Key Insights and Results

The benchmark acts as a "stress test" for modern AI. While models score above 70% on standard benchmarks like VQA v2, their performance plummeted to approximately 25-26% on OK-VQA.

  • Knowledge Retrieval is the Bottleneck: Improving visual features yielded only marginal gains. The primary challenge lies in retrieval and reasoning.
  • Category Variance: Models performed best in categories like Weather and Climate (39.84%) but struggled in Geography, History, and Culture (20.71%), where knowledge is more specific.
  • The "10-Year-Old" Test: Analysis showed that while 78% of standard VQA questions can be answered by a 10-year-old, OK-VQA is significantly more intellectually demanding.

Conclusion

OK-VQA highlights that current AI is still poor at "linking" what it sees to what it knows. While adding retrieval components like ArticleNet provides a boost, the gap remains large. This dataset opens new avenues for research, moving the goalposts from simple object recognition toward genuine cognitive understanding and open-world reasoning.

Summary updated

Abstract

Visual Question Answering (VQA) in its ideal form lets us study reasoning in the joint space of vision and language and serves as a proxy for the AI task of scene understanding. However, most VQA benchmarks to date are focused on questions such as simple counting, visual attributes, and object detection that do not require reasoning or knowledge beyond what is in the image. In this paper, we address the task of knowledge-based visual question answering and provide a benchmark, called OK-VQA, where the image content is not sufficient to answer the questions, encouraging methods that rely on external knowledge resources. Our new dataset includes more than 14,000 questions that require external knowledge to answer. We show that the performance of the state-of-the-art VQA models degrades drastically in this new setting. Our analysis shows that our knowledge-based VQA task is diverse, difficult, and large compared to previous knowledge-based VQA datasets. We hope that this dataset enables researchers to open up new avenues for research in this domain. See http://okvqa.allenai.org to download and browse the dataset.
BibTeX
@inproceedings{marino2019ok,
  author = {Kenneth Marino and Mohammad Rastegari and Ali Farhadi and Roozbeh Mottaghi},
  title = {{OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge}},
  booktitle = {CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=HobDHXXxdpS},
}
Figure