Disentangling Knowledge-based and Visual Reasoning by Question Decomposition in KB-VQA

Elham J. Barezi, Parisa KordjamshidiarXiv 2024 · DOI · arXiv

TL;DR

This paper tackles the challenge of complex, multi-hop questions in Knowledge-Based Visual Question Answering (KB-VQA) by decomposing them into simpler sub-questions. This approach allows for more targeted extraction of visual and external knowledge, leading to improved accuracy on challenging KB-VQA datasets.

What Problem Does This Paper Address?

Visual Question Answering (VQA) models are designed to answer questions about images. Knowledge-Based VQA (KB-VQA) extends this by requiring models to also leverage external knowledge beyond what's directly visible in the image. While recent advancements have incorporated Large Language Models (LLMs) to access this external knowledge, current methods struggle with complex, multi-hop questions that require reasoning across multiple pieces of information. A key limitation is that image captioning models, often used to verbalize image content for LLMs, are not adept at capturing the nuanced details needed for such complex reasoning, often focusing only on salient objects and missing crucial information for multi-hop questions. Furthermore, these complex questions might require information from both visual and non-visual sources, which current captioning-centric approaches don't effectively disentangle.

Methods

The core innovation of this paper is a question decomposition strategy. Instead of feeding a complex question directly to an LLM or captioner, the authors first break it down into a series of simpler, more focused questions. This decomposition is performed using an LLM (specifically, GPT-3.5-turbo-0125).

Following decomposition, the system performs type-checking on each sub-question to determine if it requires visual information or external knowledge.

  • Visual questions are then processed by a question-dependent explainer (like PromptCap or InstructBlip) to generate a more targeted and relevant description of the image content related to that specific sub-question.
  • Non-visual questions are answered by querying an LLM for external knowledge.

Finally, all the extracted information (decomposed visual explanations and external knowledge) is combined to form a richer context, which is then fed to an LLM to generate the final answer. The authors also incorporate Optical Character Recognition (OCR) to extract text directly from images, further enriching the visual context, especially for questions that rely on textual information within the image.

Datasets

The proposed method is evaluated on three well-known KB-VQA datasets:

  • OK-VQA: A commonly used dataset with manually filtered questions requiring external knowledge.
  • A-OKVQA: A larger dataset designed to test reasoning ability and multi-hop questions.
  • KRVQA: The largest KB-VQA dataset, featuring questions with supporting reasons indicating visual and external knowledge components.

Key Insights and Ideas

  • Decomposition is key for complexity: Breaking down complex questions into simpler ones significantly improves the ability to extract relevant visual and external information, especially for multi-hop reasoning.
  • Targeted captioning matters: Question-dependent captioners, guided by sub-questions, provide richer and more accurate visual descriptions than general captioners.
  • Disentangling visual and non-visual needs: Explicitly identifying whether a sub-question requires visual or external knowledge allows for the appropriate use of specialized tools (captioners vs. LLM knowledge retrieval).
  • OCR adds crucial context: Extracting text directly from images is vital for certain questions and can significantly boost performance, particularly on datasets with more complex visual reasoning requirements.

Key Results

The proposed method achieved up to a 2% improvement in accuracy across the evaluated datasets (A-OKVQA, OKVQA, and KRVQA) compared to state-of-the-art methods. Specifically, using InstructBlip as the question-dependent explainer generally yielded slightly better results than PromptCap. The ablation study showed that adding OCR tokens improved performance, especially for OKVQA, while question decomposition also contributed positively, with InstructBlip showing a stronger gain than PromptCap.

Limitations and Future Work

The authors acknowledge that the current chat-based captioners used for type-checking and information extraction are not as robust as specialized models like InstructBlip. They suggest that improving these chat-based captioners could further enhance the model's performance. Additionally, while they leverage LLMs for implicit knowledge, they believe that explicitly integrating external knowledge retrieval mechanisms could lead to even greater improvements. Future work could focus on training stronger visual elements for OCR perception and explanation, as well as exploring explicit knowledge retrieval alongside LLM-based implicit knowledge.

Summary updated

BibTeX
@misc{barezi2024disentangling,
  author = {Elham J. Barezi and Parisa Kordjamshidi},
  title = {{Disentangling Knowledge-based and Visual Reasoning by Question Decomposition in KB-VQA}},
  year = {2024},
  eprint = {2406.18839},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.18839},
}
Figure