TL;DR
This paper introduces PLRH, a novel framework for knowledge-based Visual Question Answering (VQA) that leverages Large Language Models (LLMs). Instead of directly prompting LLMs for answers, PLRH first guides them to generate "rationale heuristics" – intermediate reasoning steps – which are then used to inform the final answer prediction, leading to improved performance.
What Problem Does This Paper Address?
Visual Question Answering (VQA) is a challenging task that requires understanding both images and external knowledge to answer questions. Knowledge-based VQA specifically demands access to and integration of information beyond what's directly visible in an image. While recent advancements have seen the successful application of Large Language Models (LLMs) to this domain, existing methods often prompt LLMs to directly output an answer. The authors argue that this approach fails to fully utilize the LLMs' reasoning capabilities. They propose that by explicitly generating and incorporating intermediate "thought processes" or "rationales," LLMs can be better guided to produce more accurate answers, especially in complex knowledge-intensive VQA tasks.
Methods
The proposed framework, PLRH (Prompting LLMs with Rationale Heuristics), operates in three stages:
- Stage 1: Generating Rationales for Training Samples: For the training data, the authors manually craft in-context examples. These examples, along with the image caption, question, and answer, are fed to an LLM to generate corresponding "rationales" (i.e., explanations or reasoning steps). This stage essentially teaches the LLM how to produce rationales for given VQA instances.
- Stage 2: Rationale Heuristics Generation for Test Input: For unseen test data, a vanilla VQA model is used to select relevant in-context examples from the training set. Then, similar to Stage 1, the LLM is prompted with the context (image caption) and question to generate a "rationale heuristic" for the test input. Crucially, at this stage, the LLM is not given the answer.
- Stage 3: Prompting the LLM with Rationale Heuristics: In the final stage, the generated rationale heuristic for the test input is combined with the selected in-context examples (which now include their rationales and answers) and the test input itself. This comprehensive prompt is then fed to the LLM to predict the final answer. This approach aims to leverage the LLM's reasoning ability by providing it with explicit intermediate steps.
The core novelty lies in the explicit generation and utilization of these "rationale heuristics" as an intermediate step, inspired by Chain of Thought (CoT) prompting, to guide the LLM's final answer prediction.
Datasets
The paper evaluates its approach on two commonly used knowledge-based VQA benchmarks:
- OK-VQA: Contains approximately 9K training and 5K testing image-question pairs, with ten open-ended answers per sample.
- A-OKVQA: Features 17K training, 1K validation, and 7K testing samples, also with ten open-ended answers per sample. The evaluation on A-OKVQA uses direct answer evaluation on the validation set.
Key Insights and Ideas
- The Power of Intermediate Reasoning: The central idea is that LLMs benefit significantly from explicit reasoning steps. Simply asking for an answer bypasses the LLM's potential to "think aloud," which is crucial for complex tasks.
- Rationale Heuristics as Guides: The generated rationales act as "hints" or "guides" for the LLM, helping it to focus its reasoning and leverage its knowledge more effectively.
- Bridging the Gap: PLRH effectively bridges the gap between image understanding and knowledge retrieval by using rationales to connect the visual input with the external knowledge required for answering.
- CoT for Knowledge-Based VQA: The work demonstrates the applicability and effectiveness of Chain of Thought (CoT) principles in the context of knowledge-based VQA, extending its benefits beyond general reasoning tasks.
Key Results
The proposed PLRH framework achieves state-of-the-art results on both OK-VQA and A-OKVQA datasets. Specifically, it outperforms existing baselines by:
- More than 2.2% on OK-VQA.
- More than 2.1% on A-OKVQA.
The paper also includes an ablation study showing that adding the rationale heuristic consistently improves performance across different baseline setups, underscoring its importance. Figure 3 further illustrates that performance generally increases with the number of in-context examples, peaking at 8 examples for their method.
Limitations and Future Work
- Misleading Rationales: The case study (Figure 4) reveals that while rationales often help, they can sometimes mislead the LLM, leading to incorrect answers. This suggests that the quality and relevance of the generated rationales are critical.
- Future Work: The authors state their intention to continue studying LLM-based methods for knowledge-based VQA, implying further exploration into improving rationale generation and integration. They also mention the potential for further research into optimizing the number of in-context examples.
Summary updated