Introduction
Knowledge-intensive Visual Question Answering (VQA) tasks require models to reason over external information that is not present in the image itself. Standard Retrieval-Augmented Generation (RAG) frameworks typically employ a "retrieve-then-read" pipeline. However, this one-shot approach often falls short for complex queries. The authors identify three primary challenges: insufficient initial retrieval, error propagation through unstructured chat histories, and modality bias where textual or visual distractors mislead the model.
The PMSR Framework
To address these issues, the authors propose Progressive Multimodal Search and Reasoning (PMSR). Unlike static RAG, PMSR operates in an iterative loop, progressively refining its understanding through a structured reasoning trajectory.
The framework consists of four core components:
- Dual-scope Query Formulation: The model generates two types of queries at each step: record-level queries for local refinement and trajectory-level queries to maintain the broader search intent.
- Joint Search across Heterogeneous KBs: PMSR simultaneously queries textual knowledge bases (e.g., Wikipedia) and multimodal knowledge bases (image-text pairs), ensuring a holistic retrieval process.
- Compositional Reasoning: Instead of accumulating raw search results, the MLLM synthesizes new evidence into compact "reasoning records." This prevents context window bloat and keeps the reasoning chain focused.
- Adaptive Termination: An information saturation metric determines when the search has yielded sufficient information, optimizing computational efficiency.
Experimental Evaluation
The PMSR framework was evaluated across six diverse benchmarks, including InfoSeek and Encyclopedic-VQA (E-VQA) for fine-grained entity knowledge, and OK-VQA for common-sense reasoning.
Key Results
PMSR demonstrated significant improvements in both retrieval recall and end-to-end accuracy. On the InfoSeek benchmark, the system achieved a cumulative recall of 94.6%, substantially outperforming the previous state-of-the-art of 86.4%.
As shown in the results table, PMSR using the Gemini-2.5-Flash backbone achieved a record 59.9% accuracy on the challenging E-VQA single-hop benchmark, representing a nearly 9% improvement over specialized prior models. Furthermore, the adaptive termination strategy successfully reduced the average number of iterations from 5.0 to 3.3 without sacrificing performance.
Discussion and Limitations
The success of PMSR highlights the value of structured memory over raw context dumping. By maintaining a trajectory of synthesized records, the model can effectively self-correct and avoid the "lost in the weeds" phenomenon common in long-context agentic models.
However, challenges remain. The iterative nature of the framework introduces computational overhead, currently averaging 17 seconds per iteration. Additionally, the system's performance is inherently tied to the quality of the underlying retriever and the reasoning capabilities of the MLLM backbone. Future work may involve developing MLLM-based retrievers with fused multimodal embeddings to further enhance retrieval precision.
Conclusion
Progressive Multimodal Search and Reasoning (PMSR) sets a new standard for knowledge-intensive VQA by transforming retrieval into a self-refining, iterative process. By combining dual-scope queries with structured trajectory management, PMSR overcomes the limitations of traditional RAG, offering a robust path forward for complex multimodal reasoning.
Summary updated