Multimodal Commonsense Knowledge Distillation for Visual Question Answering

Shuo Yang, Siwen Luo, Soyeon Caren Han

Introduction

Visual Question Answering (VQA) has transitioned from basic visual recognition to complex reasoning tasks that demand external knowledge. While Multimodal Large Language Models (MLLMs) show promise, they are often computationally expensive to fine-tune and can struggle with structured reasoning. This paper proposes a novel framework that injects external commonsense knowledge into VQA models through knowledge distillation. By representing images, questions, and knowledge triplets as a unified relational graph, the authors enable smaller "student" models to achieve significant performance gains on the ScienceQA benchmark without the overhead of massive multimodal architectures.

Methods

The core of the proposed approach is a Teacher-Student distillation framework centered on a multimodal relational graph.

Overview of the Multimodal Commonsense Knowledge Distillation framework.

Graph Construction

For each question, a heterogeneous subgraph is constructed containing:

  • Content Nodes: These represent the question text, language context, visual context (image), and a fused vision-language feature.
  • Commonsense Nodes: Relevant knowledge triplets are retrieved from the ATOMIC2020 dataset. The system uses cosine similarity to identify the top $K=3$ most relevant knowledge snippets for each content node.

The GCN Teacher and Distillation

A two-layer Graph Convolutional Network (GCN) serves as the "teacher." Unlike standard models that simply concatenate features, the GCN explicitly models the relationships between visual objects, text, and injected commonsense facts. The knowledge captured by the GCN is then distilled into a "student" model—which can range from a simple MLP to a large Vision-Language Pretrained Model (VLPM)—using Kullback-Leibler (KL) Divergence loss. This "plug-and-play" design allows the student to inherit reasoning capabilities without performing heavy graph computations during inference.

Experimental Setup

The framework was evaluated on the ScienceQA benchmark, which includes multi-modal questions across natural science (NAT), social science (SOC), and language science (LAN). External commonsense knowledge was sourced from ATOMIC2020, a large-scale atlas of social and physical commonsense relations.

Key Results

The experimental results demonstrate consistent improvements across three tiers of student models.

Comparison of baseline models versus the proposed teacher-distilled versions across different model sizes.

As shown in the table above, the framework provides substantial boosts:

  • Small-sized (MLP): Performance increased from 42.71% to 53.92%, an 11.21% absolute improvement.
  • Medium-sized (Transformer): Improved from 48.35% to 56.79%.
  • Large-sized (VLPMs): Even state-of-the-art models like UnifiedQA saw a boost, reaching 72.33% from a 70.12% baseline.

These results suggest that relational learning is a critical missing piece even for large-scale models, and that structured knowledge distillation is an effective way to bridge this gap.

Conclusion

This work highlights the importance of relational learning in VQA. By explicitly modeling the connections between visual data and external commonsense knowledge, the proposed GCN-based teacher effectively guides student models toward better reasoning. Future work may involve refining knowledge retrieval mechanisms to reduce noise and exploring more advanced graph architectures like Graph Attention Networks to capture subtler nuances in multimodal data.

Summary updated

Abstract

Existing Multimodal Large Language Models (MLLMs) and Visual Language Pretrained Models (VLPMs) have shown remarkable performances in the general Visual Question Answering (VQA). However, these models struggle with VQA questions that require external commonsense knowledge due to the challenges in generating high-quality prompts and the high computational costs of fine-tuning. In this work, we propose a novel graph-based multimodal commonsense knowledge distillation framework that constructs a unified relational graph over commonsense knowledge, visual objects and questions through a Graph Convolutional Network (GCN) following a teacher-student environment. This proposed framework is flexible with any type of teacher and student models without further fine-tuning, and has achieved competitive performances on the ScienceQA dataset.
BibTeX
@inproceedings{yang2025multimodal,
  author = {Shuo Yang and Siwen Luo and Soyeon Caren Han},
  title = {{Multimodal Commonsense Knowledge Distillation for Visual Question Answering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=CPRDAd5lpP},
}
Figure