Multimodal Iterative RAG for Knowledge-Intensive Visual Question Answering

Changin Choi, Wonseok Lee, Jungmin Ko, Wonjong Rhee

Introduction

Knowledge-intensive Visual Question Answering (VQA) tasks require models to reason over external information that is not present in the image itself. Standard Retrieval-Augmented Generation (RAG) frameworks typically employ a "retrieve-then-read" pipeline. However, this one-shot approach often falls short for complex queries. The authors identify three primary challenges: insufficient initial retrieval, error propagation through unstructured chat histories, and modality bias where textual or visual distractors mislead the model.

The PMSR Framework

To address these issues, the authors propose Progressive Multimodal Search and Reasoning (PMSR). Unlike static RAG, PMSR operates in an iterative loop, progressively refining its understanding through a structured reasoning trajectory.

The PMSR framework architecture, illustrating the iterative loop of dual-scope query generation, joint search, and compositional reasoning.

The framework consists of four core components:

  1. Dual-scope Query Formulation: The model generates two types of queries at each step: record-level queries for local refinement and trajectory-level queries to maintain the broader search intent.
  2. Joint Search across Heterogeneous KBs: PMSR simultaneously queries textual knowledge bases (e.g., Wikipedia) and multimodal knowledge bases (image-text pairs), ensuring a holistic retrieval process.
  3. Compositional Reasoning: Instead of accumulating raw search results, the MLLM synthesizes new evidence into compact "reasoning records." This prevents context window bloat and keeps the reasoning chain focused.
  4. Adaptive Termination: An information saturation metric determines when the search has yielded sufficient information, optimizing computational efficiency.

Experimental Evaluation

The PMSR framework was evaluated across six diverse benchmarks, including InfoSeek and Encyclopedic-VQA (E-VQA) for fine-grained entity knowledge, and OK-VQA for common-sense reasoning.

Key Results

PMSR demonstrated significant improvements in both retrieval recall and end-to-end accuracy. On the InfoSeek benchmark, the system achieved a cumulative recall of 94.6%, substantially outperforming the previous state-of-the-art of 86.4%.

Comparison of PMSR against baseline models on InfoSeek and E-VQA benchmarks, highlighting the superior performance of the Gemini-2.5-Flash backbone.

As shown in the results table, PMSR using the Gemini-2.5-Flash backbone achieved a record 59.9% accuracy on the challenging E-VQA single-hop benchmark, representing a nearly 9% improvement over specialized prior models. Furthermore, the adaptive termination strategy successfully reduced the average number of iterations from 5.0 to 3.3 without sacrificing performance.

Discussion and Limitations

The success of PMSR highlights the value of structured memory over raw context dumping. By maintaining a trajectory of synthesized records, the model can effectively self-correct and avoid the "lost in the weeds" phenomenon common in long-context agentic models.

However, challenges remain. The iterative nature of the framework introduces computational overhead, currently averaging 17 seconds per iteration. Additionally, the system's performance is inherently tied to the quality of the underlying retriever and the reasoning capabilities of the MLLM backbone. Future work may involve developing MLLM-based retrievers with fused multimodal embeddings to further enhance retrieval precision.

Conclusion

Progressive Multimodal Search and Reasoning (PMSR) sets a new standard for knowledge-intensive VQA by transforming retrieval into a self-refining, iterative process. By combining dual-scope queries with structured trajectory management, PMSR overcomes the limitations of traditional RAG, offering a robust path forward for complex multimodal reasoning.

Summary updated

BibTeX
@inproceedings{choi2026multimodal,
  author = {Changin Choi and Wonseok Lee and Jungmin Ko and Wonjong Rhee},
  title = {{Multimodal Iterative RAG for Knowledge-Intensive Visual Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.881},
}
Figure