Abstract
One of the most challenging question types in Visual Question Answering (VQA) is when answering the question requires outside knowledge not present in the image. In this work, we study open-domain knowledge, the setting when the knowledge required to answer a question is not given/annotated, neither at training nor test time. We tap into two types of knowledge representations and reasoning. First, implicit knowledge which can be learned effectively from unsupervised language pre-training and supervised training data with transformer-based models. Second, explicit, symbolic knowledge encoded in knowledge bases. Our approach combines both—exploiting the powerful implicit reasoning of transformer models for answer prediction, and integrating symbolic representations from a knowledge graph, while never losing their explicit semantics to an implicit embedding. We combine diverse sources of knowledge to cover the wide variety of knowledge needed to solve knowledge-based questions. We show our approach, KRISP (Knowledge Reasoning with Implicit and Symbolic rePresentations), significantly outperforms state-of-the-art on OK-VQA, the largest available dataset for open-domain knowledge-based VQA. We show with extensive ablations that while our model successfully exploits implicit knowledge reasoning, the symbolic answer module which explicitly connects the knowledge graph to the answer vocabulary is critical to the performance of our method and generalizes to rare answers.
Introduction: The Challenge of Open-Domain Knowledge
Standard Visual Question Answering (VQA) models are typically "closed-book," trained to find answers based solely on patterns present in training images and questions. However, real-world scenarios often demand "outside knowledge." For instance, identifying a liquid as coming from a citrus fruit requires knowing that an orange is a citrus fruit—a fact not contained within the image pixels.
The challenge in Open-Domain Knowledge-Based VQA is that the required knowledge is not provided at test time. Previous attempts either attempted to learn everything implicitly (storing facts in neural network weights) or converted symbolic facts into opaque embeddings too early, losing their semantic structure. KRISP addresses this by bridging the gap between the broad, "fuzzy" reasoning of transformers and the precise, structured facts found in massive knowledge bases.
Methodology: Bridging Neural Intuition and Symbolic Facts
The KRISP architecture is a dual-stream system that merges two distinct forms of intelligence: implicit and symbolic knowledge.
Implicit Knowledge (The Transformer Branch)
The model utilizes a multi-modal BERT (MMBERT) branch. This component is pre-trained on massive text corpora, such as BooksCorpus and Wikipedia, to capture common sense and linguistic nuances. It processes image features (extracted via Faster R-CNN) and question tokens to predict an answer based on learned statistical associations.
Symbolic Knowledge (The Graph Branch)
This core innovation involves an integrated Knowledge Graph (KG) built from four sources: DBPedia, ConceptNet, VisualGenome, and a specialized hasPart KB. To make the graph relevant to a specific image, the model uses "visual symbols"—detectors for 4,000 concepts (objects, places, parts) that serve as starting nodes. Reasoning is performed using a Relational Graph Convolutional Network (RGCN) to propagate information. Crucially, KRISP maintains the symbolic identity of the nodes throughout the process, preventing the "smearing" of semantics into a generic vector space.
Integration via Late Fusion
KRISP employs a late-fusion strategy where both branches make independent predictions across the answer vocabulary. The final answer is determined by the highest confidence score from either branch. This approach allows the model to leverage the strengths of both systems: the transformer's common sense and the graph's factual precision.
Experimental Results and Insights
KRISP was evaluated on OK-VQA (Outside Knowledge VQA), the largest benchmark for this task. The model achieved a new state-of-the-art accuracy of 38.35% on OK-VQA v1.0, significantly outperforming previous models like ConceptBERT (33.66%).
The Power of the Long Tail
A key finding is that implicit models struggle with rare facts (e.g., "Who invented the steam engine?"). Symbolic graphs are far more efficient at retrieving these "long-tail" facts. KRISP predicted 1,349 unique answers compared to the baseline's 1,247, demonstrating a superior ability to identify specific entities rather than relying on common-word guesses.
Ablation and Generalization
Ablation studies confirmed that removing the symbolic graph caused a performance drop of approximately 3%, proving that the graph provides value beyond what a massive language model like BERT already contains. Furthermore, preserving symbolic identities allows the model to share reasoning logic across different symbols (e.g., applying the logic for "is a fruit" to both common and rare exotic fruits).
Limitations and Future Outlook
Despite its success, KRISP faces challenges with homonym confusion, where words with multiple meanings (e.g., "navy" as a color vs. a military branch) share the same node in the graph. Additionally, the system is limited by the coverage of its integrated KG and the accuracy of its visual detectors. Future research may focus on dynamic graph construction based on question context and more sophisticated word sense disambiguation to further enhance open-domain reasoning.
Summary updated