Query and Attention Augmentation for Knowledge-Based Explainable Reasoning

Yifeng Zhang, Ming Jiang, Qi ZhaoCVPR 2022 · DOI

Abstract

Visual Question Answering (VQA) often requires models to integrate visual information with external common-sense knowledge. Traditional Neural Module Networks (NMNs) frequently struggle with static query generation and the isolated processing of visual and factual data. We present Query and Attention Augmentation (QAA), a framework that employs reinforcement learning to generate context-aware knowledge-base queries and a memory-sharing mechanism to unify visual and external reasoning paths. Evaluated on benchmarks such as OK-VQA and FVQA, QAA demonstrates significant performance gains and provides interpretable reasoning steps.

Introduction

Standard VQA models often suffer from a "shortcut" problem, relying on statistical correlations in text rather than genuine reasoning. While Neural Module Networks (NMNs) attempt to address this by decomposing questions into functional programs, they typically face two major limitations: static queries that ignore image context and a lack of integration between visual features and external facts.

Overview of the QAA framework compared to standard reasoning.

As illustrated in Figure 1, standard approaches might fail to link a visual object (a bus) to its functional use (transport) if the query is based solely on the question text. QAA addresses this by augmenting both the queries sent to external knowledge bases and the attention mechanisms used to process visual and factual data.

Methodology: Query and Attention Augmentation

The QAA framework introduces two core components designed to bridge the gap between vision and external knowledge.

Knowledge-Based Query Augmentation (KQ)

Instead of relying on fixed keywords from the question, KQ utilizes Reinforcement Learning (RL) agents to select optimal "query concepts" from a vast vocabulary of approximately 100,000 terms. These agents learn to refine queries based on the visual context. For instance, seeing a bus in the image prompts the agent to include "transport" or "road" in its search, facilitating more accurate retrieval from external knowledge graphs like ConceptNet or WordNet.

Memory-Augmented Attention (MA)

Reasoning is an iterative process. MA implements a memory-sharing mechanism where intermediate results from visual reasoning inform the external knowledge path, and vice versa. This tight coupling ensures that as the model identifies objects in an image, it simultaneously narrows down relevant external facts, preventing the error propagation common in sequential pipelines.

The QAA architecture featuring the Program Generator and Program Executor.

Experimental Results

The model was evaluated across several key VQA benchmarks, demonstrating its robustness and generalizability:

  • OK-VQA & FVQA: QAA achieved state-of-the-art results, reaching 29.24% accuracy on OK-VQA and a significant 68.74% on FVQA. It showed particular strength in categories like "Science and Technology" and "Weather."
  • Generalization: When applied to baseline architectures like NSM and XNM, QAA consistently provided a 2-4% performance boost across different datasets, including GQA and VQA v2.
  • Explainability: By outputting augmented queries and attention maps, the model provides a transparent trace of its reasoning process, allowing users to verify the logic behind its answers.

Conclusion

QAA represents a significant step toward more robust and explainable visual reasoning. By interleaving knowledge retrieval with visual processing through RL-based query augmentation and shared memory attention, the framework overcomes the limitations of static, siloed reasoning models. While the model remains dependent on the quality of its external knowledge bases, its ability to perform joint reasoning offers a powerful template for future multimodal AI research.

Summary updated

BibTeX
@inproceedings{zhang2022query,
  author = {Yifeng Zhang and Ming Jiang and Qi Zhao},
  title = {{Query and Attention Augmentation for Knowledge-Based Explainable Reasoning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.01513},
}
Figure