Abstract
Thanks to the strong reasoning capabilities of Large Language Models (LLMs), recent approaches to knowledge-based visual question answering (KVQA) utilize LLMs with a global caption of an input image to answer a question. However, these approaches may miss key visual information that is not captured by the caption. Moreover, they cannot fully utilize the visual information required to answer the question. To address these issues, we introduce a new framework called Multi-Modal Knowledge-Aware Reasoner (MM-Reasoner) for KVQA. MM-Reasoner first utilizes a set of vision APIs, such as dense captioners, object detectors, and OCR, to extract detailed information from the image in textual format. Then, it prompts an LLM to extract query-specific knowledge from the extracted textual information to provide a rich representation that contains external knowledge, commonsense, explicit supporting facts, and rationales required for reasoning. Finally, the knowledge, query, and visual input are used to fine-tune a Vision-Language Model (VLM). At test time, MM-Reasoner uses the potential answers predicted by the VLM to iteratively update and optimize the prompt, refining its answer. Empirical studies show that MM-Reasoner achieves state-of-the-art performance on several KVQA datasets.
Introduction
Knowledge-Based Visual Question Answering (KVQA) requires a model to reason using both visual information and external knowledge. For instance, answering "Who sang the song referenced on this sign?" requires identifying text via OCR and then accessing musical history knowledge. Current approaches often suffer from an "information gap": LLMs are knowledgeable but lack direct visual access beyond simple captions, while Vision-Language Models (VLMs) perceive pixels well but lack the vast external knowledge of LLMs. MM-Reasoner bridges this gap by combining the perceptual precision of vision APIs and VLMs with the reasoning power of LLMs.
Methodology
MM-Reasoner employs a three-stage pipeline to synthesize visual data and external knowledge:
- Vision APIs: The framework uses specialized APIs (Dense Captioners, OCR, Object Detectors, Celebrity Recognizers) to convert the image into a structured textual representation, capturing fine-grained details often missed by global captions.
- LLM Prompting: This textual data and the question are fed into an LLM (e.g., GPT-4). The LLM generates rationales—supporting facts and logical steps—to bridge the gap between visual evidence and the final answer.
- VLM Fine-Tuning: A VLM (like i-Code v2 or Flamingo) is fine-tuned using the original image, the question, and the LLM-generated rationales. This allows the model to ground the LLM's logic in the actual image pixels.
A key innovation is Iterative Reasoning at test time. The model uses the VLM’s initial prediction to update the LLM prompt, allowing the LLM to refine its rationales in a closed-loop feedback system.
Evaluation and Results
The framework was evaluated on four major benchmarks: OK-VQA, A-OKVQA, FVQA, and KVQA. MM-Reasoner achieved state-of-the-art (SOTA) performance across all datasets:
- OK-VQA: 60.8% accuracy, outperforming previous leaders like PromptCap and REVEAL.
- A-OKVQA: 60.2% accuracy.
- FVQA: 61.1% accuracy, surpassing human-annotated baselines.
- Few-Shot Learning: On OK-VQA, it outperformed Flamingo-9B by 4.3 points in low-data scenarios.
Key Insights and Discussion
The success of MM-Reasoner highlights several critical points in multi-modal reasoning. First, textual descriptions alone are insufficient; the VLM's ability to process raw pixels is necessary for questions requiring fine-grained visual attributes. Second, rationales serve as an effective bridge, providing the VLM with structured "hints" rather than raw data.
While powerful, the framework currently relies on frozen LLMs and external API accuracy. Future work may involve fine-tuning the LLM specifically for rationale generation or integrating real-time tools like web search to expand the model's knowledge base further.
Summary updated