MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answering

Xianwei Mao, Kai Ye, Sheng Zhou, Nan Zhang, Haikuan Huang, Bin Li, Jiajun BuarXiv 2026 · DOI

TL;DR

This paper introduces MaS-VQA, a novel framework that intelligently filters both visual and textual information to improve knowledge-based visual question answering (KB-VQA). By "masking" irrelevant parts and "selecting" key pieces, MaS-VQA helps large language models reason more effectively with external knowledge, leading to better accuracy on complex questions.

What Problem Does This Paper Address?

Visual Question Answering (VQA) systems are great at answering questions based solely on an image and its accompanying text. However, many real-world questions require external knowledge – like facts from Wikipedia or commonsense reasoning. This is where Knowledge-Based VQA (KB-VQA) comes in, aiming to combine visual understanding with external information.

The challenge is that the external knowledge retrieved is often noisy, partially irrelevant, or misaligned with the visual content. Simply dumping all this information into a model can confuse it, leading to inaccurate answers. Existing methods struggle to effectively control which visual and which retrieved knowledge are used together for reasoning. This paper tackles this problem by proposing a framework that actively filters and selects relevant information before feeding it to the model.

Methods

MaS-VQA introduces a "Mask-and-Select" framework that tightly integrates explicit knowledge filtering with implicit knowledge reasoning. Here's how it works:

  1. Multimodal Retrieval: First, the system retrieves a set of top-k candidate knowledge passages related to the image and question.
  2. Mask-and-Select (Explicit Knowledge Processing): This is the core innovation. It involves two parts:
  3. Knowledge-Guided Attention Mask (Visual): A module creates a mask that highlights relevant image regions by considering both the question and the retrieved text. This helps suppress irrelevant visual areas.
  4. Question-Conditioned Phrase Selection (Textual): This module prunes noisy or weakly relevant parts of the retrieved text, preserving only key knowledge fragments. These two steps work together to create a compact, high-signal "explicit knowledge package" (combining filtered visual and textual information).

  5. Implicit Knowledge Processing: The filtered explicit knowledge is then used to guide a frozen large multimodal language model (MLLM). The MLLM generates a concise "implicit knowledge paragraph" that summarizes grounded insights, effectively distilling useful information and bridging gaps.

  6. Answering: Finally, the MLLM uses the original image, question, the explicit knowledge package, and the generated implicit knowledge paragraph to predict the final answer.

This approach contrasts with standard methods that might separate explicit and implicit knowledge reasoning or rely on simpler filtering. MaS-VQA aims for a more integrated and controlled co-modeling of both knowledge sources.

Datasets

The paper evaluates MaS-VQA on two established KB-VQA benchmarks:

  • Encyclopedic-VQA (E-VQA): A large dataset with millions of visual question-answer pairs, often requiring knowledge about fine-grained entities.
  • InfoSeek: Another large dataset focused on information retrieval-based VQA, designed to test generalization to unseen entities and questions.

Key Insights and Ideas

  • Noise is a major bottleneck: Simply retrieving knowledge isn't enough; the noise and irrelevance within retrieved information significantly hinder performance.
  • Cross-modal coupling is key: The relevance of visual information often depends on textual knowledge, and vice-versa. MaS-VQA's joint filtering addresses this interdependence.
  • Selection over aggregation: Instead of just aggregating information, actively selecting and filtering the most relevant parts (both visual and textual) leads to more robust reasoning.
  • Implicit knowledge benefits from explicit guidance: Generating implicit knowledge from a frozen MLLM is more effective when guided by carefully filtered explicit evidence. This helps the MLLM focus its internal knowledge.

Key Results

MaS-VQA consistently outperforms strong baselines on both E-VQA and InfoSeek across various MLLM backbones. For example, using the Qwen3-VL-8B backbone, MaS-VQA achieved an accuracy of 42.2% on E-VQA's Single-Hop questions, a significant improvement over the zero-shot baseline of 19.5%. The framework also demonstrates strong generalization capabilities on InfoSeek. Ablation studies confirm that both the attention masking and phrase selection components are crucial for performance, and their combination yields the best results. Qualitative examples show MaS-VQA successfully grounding knowledge to images and correcting errors made by simpler methods.

Limitations and Future Work

The authors acknowledge that their method, like many in the field, could potentially amplify biases present in the underlying knowledge base or MLLM. They also note that the method doesn't guarantee factual correctness and should not be used for high-stakes decisions without further verification. Future work could focus on stronger evidence verification, uncertainty estimation, and bias mitigation for KB-VQA systems. The paper also suggests that the retrieval breadth (number of passages) has an optimal point, and further increasing it can lead to diminishing returns or even performance drops.

Summary updated

BibTeX
@misc{mao2026mas,
  author = {Xianwei Mao and Kai Ye and Sheng Zhou and Nan Zhang and Haikuan Huang and Bin Li and Jiajun Bu},
  title = {{MaS-VQA: A Mask-and-Select Framework for Knowledge-Based Visual Question Answering}},
  year = {2026},
  eprint = {2602.15915},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.15915},
}
Figure