KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA

Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, Marcus Rohrbach

Abstract

One of the most challenging question types in Visual Question Answering (VQA) is when answering requires outside knowledge not present in the image. We study open-domain knowledge-based VQA, where the required knowledge is not provided at training or test time. We propose KRISP (Knowledge Reasoning with Implicit and Symbolic rePresentations), which combines implicit knowledge from unsupervised language pretraining with explicit, symbolic knowledge from structured knowledge bases. KRISP significantly outperforms state-of-the-art models on the OK-VQA dataset by preserving explicit semantics and integrating diverse knowledge sources.

The Challenge of Outside Knowledge

Standard VQA models excel at identifying objects within an image but often stumble when a question demands information beyond the pixels. For instance, identifying a glass of orange juice is a visual task, but answering "Which liquid here comes from a citrus fruit?" requires the external fact that oranges are citrus fruits.

Example of a knowledge-based VQA task requiring both symbolic and implicit knowledge.

In open-domain knowledge-based VQA, models must draw from a vast, unannotated universe of information. Previous approaches either relied solely on implicit knowledge stored in neural weights or converted symbols into abstract embeddings too early, losing their specific semantic meaning.

The KRISP Architecture

KRISP addresses these limitations through a dual-pathway architecture that processes information through two distinct lenses:

  1. Implicit Pathway (MMBERT): A Multi-modal BERT transformer pretrained on massive text corpora (Wikipedia and BooksCorpus). It captures general linguistic "common sense" and aligns visual features with textual tokens.
  2. Symbolic Pathway (RGCN): A Relational Graph Convolutional Network operating on a massive Knowledge Graph (KG). This KG merges four diverse sources: DBPedia (encyclopedic facts), ConceptNet (common sense), VisualGenome (spatial/visual relations), and hasPart KB (scientific part-of knowledge).
The KRISP architecture featuring dual implicit and symbolic pathways with late fusion.

The "secret sauce" of KRISP is maintaining the symbolic identity of nodes throughout the graph reasoning process. Instead of squashing the graph into a generic embedding, the symbolic branch makes predictions by directly referencing nodes that correspond to potential answers. The final output is determined via late fusion, selecting the highest-scoring prediction from either branch.

Key Insights and Results

The research yields several critical insights into how models handle external knowledge:

  • Symbols are Crucial: Preserving the link between graph nodes and actual words (symbols) allows the model to reason about specific, rare facts that might be lost in a standard neural embedding.
  • Complementary Strengths: The implicit branch acts as a powerful generalist, while the symbolic branch serves as a specialist for the "long tail" of rare facts. In testing, the symbolic branch was responsible for the final prediction 47% of the time.
  • State-of-the-Art Performance: KRISP achieved 38.35% accuracy on OK-VQA v1.0, a significant improvement over the previous best of 33.66%.
  • Generalization to the Long Tail: KRISP predicted 1,349 unique correct answers, compared to 1,247 for a non-symbolic baseline, confirming its ability to handle rare, specific information.

Conclusion

KRISP demonstrates that integrating structured, symbolic knowledge with powerful implicit reasoning is key to solving open-domain VQA. By maintaining explicit semantics, the model effectively bridges the gap between visual perception and encyclopedic reasoning. Future work will focus on scaling these methods to even larger knowledge bases and deepening the integration between symbolic and implicit pathways.

Summary updated

Abstract

One of the most challenging question types in VQA is when answering the question requires outside knowledge not present in the image. In this work we study open-domain knowledge, the setting when the knowledge required to answer a question is not given/annotated, neither at training nor test time. We tap into two types of knowledge representations and reasoning. First, implicit knowledge which can be learned effectively from unsupervised language pretraining and supervised training data with transformer-based models. Second, explicit, symbolic knowledge encoded in knowledge bases. Our approach combines both---exploiting the powerful implicit reasoning of transformer models for answer prediction, and integrating symbolic representations from a knowledge graph, while never losing their explicit semantics to an implicit embedding. We combine diverse sources of knowledge to cover the wide variety of knowledge needed to solve knowledge-based questions. We show our approach, KRISP, significantly outperforms state-of-the-art on OK-VQA, the largest available dataset for open-domain knowledge-based VQA. We show with extensive ablations that while our model successfully exploits implicit knowledge reasoning, the symbolic answer module which explicitly connects the knowledge graph to the answer vocabulary is critical to the performance of our method and generalizes to rare answers.
BibTeX
@inproceedings{marino2021krisp,
  author = {Kenneth Marino and Xinlei Chen and Devi Parikh and Abhinav Gupta and Marcus Rohrbach},
  title = {{KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=zQPERowzc7O},
}
Figure