Abstract
Recently, Retrieval-Augmented Generation (RAG) has been proposed to expand internal knowledge of Multimodal Large Language Models (MLLMs) by incorporating external knowledge databases into the generation process, which is widely used for knowledge-based Visual Question Answering (VQA) tasks. Despite impressive advancements, vanilla RAG-based VQA methods that rely on unstructured documents and overlook the structural relationships among knowledge elements frequently introduce irrelevant or misleading content, reducing answer accuracy and reliability. To overcome these challenges, a promising solution is to integrate multimodal knowledge graphs (KGs) into RAG-based VQA frameworks to enhance the generation by introducing structured multimodal knowledge. Therefore, in this paper, we propose a novel multimodal knowledge-augmented generation framework (mKG-RAG) based on multimodal KGs for knowledge-intensive VQA tasks. Specifically, our approach leverages MLLM-powered keyword extraction and vision-text matching to distill semantically consistent and modality-aligned entities/relationships from multimodal documents, constructing high-quality multimodal KGs as structured knowledge representations. In addition, a dual-stage retrieval strategy equipped with a question-aware multimodal retriever is introduced to improve retrieval efficiency while refining precision. Comprehensive experiments demonstrate that our approach significantly outperforms existing methods, setting a new state-of-the-art for knowledge-based VQA.
Introduction
Multimodal Large Language Models (MLLMs) like LLaVA or GPT-4o have shown remarkable capabilities in visual understanding. However, they often struggle with "knowledge-intensive" questions—tasks requiring specific, encyclopedic facts not explicitly present in the visual input. For instance, determining the exact year a specific stadium was renovated requires external factual knowledge beyond simple image recognition.
Traditional Retrieval-Augmented Generation (RAG) attempts to solve this by searching unstructured text databases (e.g., Wikipedia). However, vanilla RAG faces significant hurdles:
- Noise and Irrelevance: Unstructured text often contains "fluff" that distracts the model.
- Structural Blindness: Standard RAG ignores logical relationships between entities, such as location or historical timelines.
- The Modality Gap: Purely textual knowledge graphs make it difficult to link visual regions in an image to specific database entries.
Methodology
To address these limitations, we introduce mKG-RAG, a framework that transforms unstructured documents into structured Multimodal Knowledge Graphs (MKGs). This approach provides a "structured memory" for MLLMs, enabling more precise and context-aware retrieval.
Multimodal KG Construction
The framework builds task-specific graphs from multimodal documents through three key steps:
- Textual Extraction: An LLM identifies entities and their relationships within the text.
- Visual Extraction: Scene Graph Generation (SGG) identifies objects and spatial relationships within images.
- Vision-Text Matching: An MLLM aligns visual regions with textual entities, creating a unified graph where nodes represent both textual descriptions and image crops.
Dual-Stage Retrieval and the QM-Retriever
Searching massive graphs efficiently requires a specialized strategy. mKG-RAG employs a two-step process:
- Coarse-grained Retrieval: A vector search identifies the top-K relevant documents.
- Fine-grained Graph Traversal: The system performs traversal on the sub-graphs of these documents to pinpoint specific entities and relationships.
Central to this process is the Question-aware Multimodal Retriever (QM-Retriever). Unlike standard retrievers that focus on simple similarity, the QM-Retriever uses a "Question Converter" to reformulate interrogative queries into declarative statements in the latent space. This aligns the query more effectively with factual evidence in the database.
Experimental Results
The framework was evaluated on two major benchmarks: E-VQA (linked to 2 million Wikipedia pages) and InfoSeek (focused on "long-tail" niche facts). mKG-RAG established a new state-of-the-art across these datasets:
- E-VQA: Achieved 36.3% accuracy, a significant improvement over zero-shot MLLMs (~16%) and previous RAG methods like ReflectiVA (29.2%).
- InfoSeek: Reached 40.5% accuracy, outperforming existing models by a healthy margin.
- Generalization: The performance boost was consistent across various model sizes, from 3B to 32B parameters, with an average improvement of approximately 9%.
Discussion and Conclusion
The success of mKG-RAG highlights that structure filters noise. By distilling documents into a graph, the model avoids linguistic distractions that often lead to hallucinations. Furthermore, the use of "l-hop" graph traversal ensures the model receives broader context, such as the specific event associated with a renovation date.
While the current framework focuses on binary relationships, future work will explore more complex "hyper-relationships." Additionally, while graph construction is performed offline, managing the computational overhead of real-time graph traversal and adapting to dynamic, ever-changing knowledge bases remain key areas for future development.
Summary updated