MM-Reasoner: A Multi-Modal Knowledge-Aware Framework for Knowledge-Based Visual Question Answering

Mahmoud Khademi, Ziyi Yang, Felipe Vieira Frujeri, Chenguang ZhuEMNLP 2023 · OpenReview

Abstract

Thanks to the strong reasoning capabilities of Large Language Models (LLMs), recent approaches to knowledge-based visual question answering (KVQA) utilize LLMs with a global caption of an input image to answer a question. However, these approaches may miss key visual information that is not captured by the caption. Moreover, they cannot fully utilize the visual information required to answer the question. To address these issues, we introduce a new framework called Multi-Modal Knowledge-Aware Reasoner (MM-Reasoner) for KVQA. MM-Reasoner first utilizes a set of vision APIs, such as dense captioners, object detectors, and OCR, to extract detailed information from the image in textual format. Then, it prompts an LLM to extract query-specific knowledge from the extracted textual information to provide a rich representation that contains external knowledge, commonsense, explicit supporting facts, and rationales required for reasoning. Finally, the knowledge, query, and visual input are used to fine-tune a Vision-Language Model (VLM). At test time, MM-Reasoner uses the potential answers predicted by the VLM to iteratively update and optimize the prompt, refining its answer. Empirical studies show that MM-Reasoner achieves state-of-the-art performance on several KVQA datasets.

Introduction

Knowledge-Based Visual Question Answering (KVQA) requires a model to reason using both visual information and external knowledge. For instance, answering "Who sang the song referenced on this sign?" requires identifying text via OCR and then accessing musical history knowledge. Current approaches often suffer from an "information gap": LLMs are knowledgeable but lack direct visual access beyond simple captions, while Vision-Language Models (VLMs) perceive pixels well but lack the vast external knowledge of LLMs. MM-Reasoner bridges this gap by combining the perceptual precision of vision APIs and VLMs with the reasoning power of LLMs.

Methodology

MM-Reasoner employs a three-stage pipeline to synthesize visual data and external knowledge:

  1. Vision APIs: The framework uses specialized APIs (Dense Captioners, OCR, Object Detectors, Celebrity Recognizers) to convert the image into a structured textual representation, capturing fine-grained details often missed by global captions.
  2. LLM Prompting: This textual data and the question are fed into an LLM (e.g., GPT-4). The LLM generates rationales—supporting facts and logical steps—to bridge the gap between visual evidence and the final answer.
  3. VLM Fine-Tuning: A VLM (like i-Code v2 or Flamingo) is fine-tuned using the original image, the question, and the LLM-generated rationales. This allows the model to ground the LLM's logic in the actual image pixels.
The MM-Reasoner framework pipeline, showing the flow from Vision APIs to LLM rationale generation and final VLM prediction.

A key innovation is Iterative Reasoning at test time. The model uses the VLM’s initial prediction to update the LLM prompt, allowing the LLM to refine its rationales in a closed-loop feedback system.

Evaluation and Results

The framework was evaluated on four major benchmarks: OK-VQA, A-OKVQA, FVQA, and KVQA. MM-Reasoner achieved state-of-the-art (SOTA) performance across all datasets:

  • OK-VQA: 60.8% accuracy, outperforming previous leaders like PromptCap and REVEAL.
  • A-OKVQA: 60.2% accuracy.
  • FVQA: 61.1% accuracy, surpassing human-annotated baselines.
  • Few-Shot Learning: On OK-VQA, it outperformed Flamingo-9B by 4.3 points in low-data scenarios.

Key Insights and Discussion

The success of MM-Reasoner highlights several critical points in multi-modal reasoning. First, textual descriptions alone are insufficient; the VLM's ability to process raw pixels is necessary for questions requiring fine-grained visual attributes. Second, rationales serve as an effective bridge, providing the VLM with structured "hints" rather than raw data.

Examples of MM-Reasoner's reasoning process, utilizing OCR for cultural references and celebrity recognition for biographical questions.

While powerful, the framework currently relies on frozen LLMs and external API accuracy. Future work may involve fine-tuning the LLM specifically for rationale generation or integrating real-time tools like web search to expand the model's knowledge base further.

Summary updated

Abstract

Thanks to the strong reasoning capabilities of Large Language Models (LLMs), recent approaches to knowledge-based visual question answering (KVQA) utilize LLMs with a global caption of an input image to answer a question. However, these approaches may miss key visual information that is not captured by the caption. Moreover, they cannot fully utilize the visual information required to answer the question. To address these issues, we introduce a new framework called Multi-Modal Knowledge-Aware Reasoner (MM-Reasoner) for KVQA. MM-Reasoner first utilizes a set of vision APIs, such as dense captioners, object detectors, and OCR, to extract detailed information from the image in textual format. Then, it prompts an LLM to extract query-specific knowledge from the extracted textual information to provide a rich representation that contains external knowledge, commonsense, explicit supporting facts, and rationales required for reasoning. Finally, the knowledge, query, and visual input are used to fine-tune a Vision-Language Model (VLM). At test time, MM-Reasoner uses the potential answers predicted by the VLM to iteratively update and optimize the prompt, refining its answer. Empirical studies show that MM-Reasoner achieves state-of-the-art performance on several KVQA datasets.
BibTeX
@inproceedings{khademi2023mm,
  author = {MAHMOUD KHADEMI and Ziyi Yang and Felipe Vieira Frujeri and Chenguang Zhu},
  title = {{MM-Reasoner: A Multi-Modal Knowledge-Aware Framework for Knowledge-Based Visual Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=c27QqxALfo},
}
Figure