Abstract
Visual Question Answering (VQA) requires reasoning across visual and textual modalities, yet Large Vision-Language Models (LVLMs) often lack integrated commonsense knowledge, limiting their robustness in real-world scenarios. To address this, we introduce MAGIC-VQA, a novel framework that enhances VQA by systematically integrating commonsense knowledge with LVLMs. MAGIC-VQA employs a three-stage process: (1) Explicit Knowledge Integration from external sources, (2) By-Type Post-Processing for contextual refinement, and (3) Implicit Knowledge Augmentation using a Graph Neural Network (GNN) for structured reasoning. While GNNs bring greater depth to structured inference, they enable superior relational inference beyond LVLMs. MAGIC-VQA bridges a key gap by unifying commonsense knowledge with LVLM-driven reasoning, eliminating the need for extensive pre-training or complex prompt tuning. Our framework achieves state-of-the-art performance on benchmark datasets, significantly improving commonsense reasoning in VQA.
Introduction: The Commonsense Gap in LVLMs
While Large Vision-Language Models (LVLMs) like GPT-4o or LLaVA excel at object recognition and basic scene description, they often stumble on questions requiring "everyday" reasoning. For instance, a model might correctly identify a person holding a trophy but fail to infer the social context (e.g., a victory celebration) or the physical properties of the objects involved.
This "commonsense gap" exists because models are primarily trained on massive image-text pairs, which capture explicit descriptions but often miss the implicit rules of the world. Previous attempts to fix this relied on expensive retraining or static prompts that don't adapt to new scenarios. MAGIC-VQA addresses this by injecting missing knowledge dynamically and efficiently, adding only 0.33M trainable parameters.
Methodology: A Three-Stage Grounding Pipeline
The MAGIC-VQA framework employs a three-stage process to ground visual reasoning in commonsense knowledge:
- Explicit Knowledge Integration: The system generates an image caption (using BLIP2) to provide extra context. It then encodes the image, question, and caption into a shared embedding space to retrieve relevant "knowledge triples" (subject-relation-object) from ATOMIC2020, a massive commonsense knowledge graph.
- By-Type Post-Processing: Retrieved triples are filtered into three categories: Physical Entity (PE) (e.g., material properties), Event-Centered (EC) (e.g., temporal sequences), and Social Interaction (SI) (e.g., emotional impact). Dynamic thresholding assigns relevance levels (High, Medium, Low) to each piece of knowledge.
- Implicit Knowledge Augmentation: The system constructs a heterogeneous multimodal graph connecting the image, question, and caption nodes to the retrieved commonsense nodes. A two-layer Graph Convolutional Network (GCN) processes this structure to generate confidence scores for candidate answers, which guide the final LVLM inference.
What makes this approach novel is the decoupling of knowledge acquisition from the model's core capacity. Instead of retraining a 7B parameter model, MAGIC-VQA utilizes a lean GNN "reasoning engine" that acts as a smart advisor to the larger model.
Key Insights and Results
Evaluation across diverse benchmarks—including ScienceQA, TextVQA, and MMMU—revealed that different tasks require specific "flavors" of commonsense. For example, ScienceQA performance peaked when emphasizing Physical Entity (PE) knowledge, while TextVQA benefited more from Event-Centered (EC) context.
The framework consistently improved performance across all tested backbones:
- ScienceQA: GPT-4o-mini’s accuracy rose from 76.45% to 81.22%.
- MMMU: Qwen2VL saw a significant jump from 51.10% to 57.42%.
- Benchmarking: MAGIC-VQA outperformed existing knowledge-augmented models like KAT and KRISP on the A-OKVQA and VCR datasets, often by double-digit margins.
Conclusion
MAGIC-VQA demonstrates that structured commonsense knowledge can be integrated into LVLMs without the need for massive compute resources. By combining explicit retrieval with implicit graph-based reasoning, the framework provides a scalable, "plug-and-play" solution for more robust visual reasoning. Future work will focus on expanding knowledge sources and improving the system's resilience to noisy or ambiguous inputs.
Summary updated