Abstract
Visual Question Answering (VQA) serves as a critical proxy for scene understanding in AI. However, existing benchmarks often focus on simple visual recognition tasks like counting or attribute detection. This paper introduces OK-VQA (Outside Knowledge Visual Question Answering), a benchmark containing over 14,000 questions that require external knowledge to answer. By shifting the focus from "visual lookup" to "open-world reasoning," OK-VQA reveals a significant performance gap in state-of-the-art models, which struggle to link visual content with unstructured knowledge sources like Wikipedia.
The Problem: The Limits of Visual Lookup
Standard VQA models have become adept at parsing pixels to answer questions like "What color is the car?" or "How many apples are in the bowl?" Yet, true scene understanding requires more than just vision; it requires external knowledge.
Consider an image of a teddy bear. A standard VQA model might identify the object, but asking "Which American president is associated with this stuffed animal?" requires the model to retrieve the historical fact that the toy was named after Theodore Roosevelt. Existing datasets often lack this requirement or rely on overly simplistic, structured knowledge triplets that fail to capture real-world complexity.
The OK-VQA Dataset
The researchers curated a dataset of 14,055 questions paired with 14,031 images from the COCO dataset. To ensure the benchmark truly tests external knowledge, the authors employed a multi-stage filtering process with human annotators to exclude questions that could be answered by visual inspection alone.
The questions are categorized into 10 diverse domains, including Science & Technology, History, and Weather & Climate.
Unlike previous efforts like FVQA, which use fixed knowledge graphs, OK-VQA encourages the use of unstructured, open-knowledge sources, forcing models to navigate the "messiness" of human information.
Methodology: Bridging Vision and Knowledge
To establish baselines, the authors tested state-of-the-art models like MUTAN and BAN. They also introduced ArticleNet (AN), a retrieval engine designed to generate search queries from questions and visual classifiers, retrieve relevant Wikipedia articles, and score sentences to identify the most likely answer. Hybrid models (e.g., MUTAN + AN) were then created to feed this retrieved knowledge back into the visual reasoning engine.
Key Insights and Results
The benchmark acts as a "stress test" for modern AI. While models score above 70% on standard benchmarks like VQA v2, their performance plummeted to approximately 25-26% on OK-VQA.
- Knowledge Retrieval is the Bottleneck: Improving visual features yielded only marginal gains. The primary challenge lies in retrieval and reasoning.
- Category Variance: Models performed best in categories like Weather and Climate (39.84%) but struggled in Geography, History, and Culture (20.71%), where knowledge is more specific.
- The "10-Year-Old" Test: Analysis showed that while 78% of standard VQA questions can be answered by a 10-year-old, OK-VQA is significantly more intellectually demanding.
Conclusion
OK-VQA highlights that current AI is still poor at "linking" what it sees to what it knows. While adding retrieval components like ArticleNet provides a boost, the gap remains large. This dataset opens new avenues for research, moving the goalposts from simple object recognition toward genuine cognitive understanding and open-world reasoning.
Summary updated