Introduction
Visual Question Answering (VQA) has transitioned from basic visual recognition to complex reasoning tasks that demand external knowledge. While Multimodal Large Language Models (MLLMs) show promise, they are often computationally expensive to fine-tune and can struggle with structured reasoning. This paper proposes a novel framework that injects external commonsense knowledge into VQA models through knowledge distillation. By representing images, questions, and knowledge triplets as a unified relational graph, the authors enable smaller "student" models to achieve significant performance gains on the ScienceQA benchmark without the overhead of massive multimodal architectures.
Methods
The core of the proposed approach is a Teacher-Student distillation framework centered on a multimodal relational graph.
Graph Construction
For each question, a heterogeneous subgraph is constructed containing:
- Content Nodes: These represent the question text, language context, visual context (image), and a fused vision-language feature.
- Commonsense Nodes: Relevant knowledge triplets are retrieved from the ATOMIC2020 dataset. The system uses cosine similarity to identify the top $K=3$ most relevant knowledge snippets for each content node.
The GCN Teacher and Distillation
A two-layer Graph Convolutional Network (GCN) serves as the "teacher." Unlike standard models that simply concatenate features, the GCN explicitly models the relationships between visual objects, text, and injected commonsense facts. The knowledge captured by the GCN is then distilled into a "student" model—which can range from a simple MLP to a large Vision-Language Pretrained Model (VLPM)—using Kullback-Leibler (KL) Divergence loss. This "plug-and-play" design allows the student to inherit reasoning capabilities without performing heavy graph computations during inference.
Experimental Setup
The framework was evaluated on the ScienceQA benchmark, which includes multi-modal questions across natural science (NAT), social science (SOC), and language science (LAN). External commonsense knowledge was sourced from ATOMIC2020, a large-scale atlas of social and physical commonsense relations.
Key Results
The experimental results demonstrate consistent improvements across three tiers of student models.
As shown in the table above, the framework provides substantial boosts:
- Small-sized (MLP): Performance increased from 42.71% to 53.92%, an 11.21% absolute improvement.
- Medium-sized (Transformer): Improved from 48.35% to 56.79%.
- Large-sized (VLPMs): Even state-of-the-art models like UnifiedQA saw a boost, reaching 72.33% from a 70.12% baseline.
These results suggest that relational learning is a critical missing piece even for large-scale models, and that structured knowledge distillation is an effective way to bridge this gap.
Conclusion
This work highlights the importance of relational learning in VQA. By explicitly modeling the connections between visual data and external commonsense knowledge, the proposed GCN-based teacher effectively guides student models toward better reasoning. Future work may involve refining knowledge retrieval mechanisms to reduce noise and exploring more advanced graph architectures like Graph Attention Networks to capture subtler nuances in multimodal data.
Summary updated