Visual Query Answering by Entity-Attribute Graph Matching and Reasoning

Peixi Xiong, Huayi Zhan, Xin Wang, Baivab Sinha, Ying Wu

Abstract

Visual Query Answering (VQA) is of great significance in offering people convenience: one can raise a question for details of objects, or high-level understanding about the scene, over an image. This paper proposes a novel method to address the VQA problem. In contrast to prior works, our method that targets single scene VQA, replies on graph-based techniques and involves reasoning. In a nutshell, our approach is centered on three graphs. The first graph, referred to as inference graph $G_I$, is constructed via learning over labeled data. The other two graphs, referred to as query graph $Q$ and entity-attribute graph $G_{EA}$, are generated from natural language query $Q_{nl}$ and image $Img$, that are issued from users, respectively. As $G_{EA}$ often does not take sufficient information to answer $Q$, we develop techniques to infer missing information of $G_{EA}$ with $G_I$. Based on $G_{EA}$ and $Q$, we provide techniques to find matches of $Q$ in $G_{EA}$, as the answer of $Q_{nl}$ in $Img$. Unlike commonly used VQA methods that are based on end-to-end neural networks, our graph-based method shows well-designed reasoning capability, and thus is highly interpretable. We also create a dataset on soccer match (Soccer-VQA) with rich annotations. The experimental results show that our approach outperforms the state-of-the-art method and has high potential for future investigation.

Introduction

Standard Visual Query Answering (VQA) systems typically utilize end-to-end deep learning architectures, where a Convolutional Neural Network (CNN) processes visual data and a Recurrent Neural Network (RNN) processes the query. While these models perform well on simple recognition tasks, they often act as "black boxes" that struggle with multi-step reasoning. Research suggests these models frequently rely on dataset biases—such as memorizing that the answer to "how many..." is usually "2"—rather than performing true scene understanding. This paper addresses the need for interpretable reasoning by proposing a framework that handles "hidden" information through structured graph matching and logical inference.

Methodology: The Three-Graph Approach

The proposed framework shifts the VQA task from a predictive guessing problem to a formal search and reasoning problem centered on three distinct graph structures:

  1. Entity-Attribute Graph ($G_{EA}$): This graph represents the visual scene. Nodes represent entities (e.g., players, the ball) while edges and node properties represent attributes like color, location, and status.
  2. Query Graph ($Q$): Natural language queries are parsed into structured graphs. For instance, "How many players are there?" is transformed into a search for entities of type "person" with the role "player."
  3. Inference Graph ($G_I$): Serving as the system's reasoning engine, this uses a Bayesian Network to infer missing or hidden attributes. For example, it can calculate the probability that a person is a "referee" based on their unique uniform color and position relative to the field.
Overview of the graph-based VQA approach, including the construction of the Entity-Attribute Graph and the Query Graph.

The final answer is derived through Graph Pattern Matching, where the system identifies subgraphs in $G_{EA}$ that match the structure of $Q$. This step-by-step process ensures high interpretability, allowing developers to pinpoint whether a failure occurred in visual detection, attribute inference, or matching.

The Soccer-VQA Dataset

To evaluate the system, the authors introduced Soccer-VQA, a benchmark consisting of 7,900 frames from 2016 FIFA World Cup footage. Soccer provides a constrained yet complex domain where roles (referee, goalkeeper, player) and rules are paramount for reasoning. The dataset is richly annotated with bounding boxes, roles, uniform colors, and field locations.

An example of a soccer scene from the Soccer-VQA dataset with detailed entity and attribute annotations.

Key Results

The graph-based approach was compared against state-of-the-art neural models, including LSTM+CNN and HieCoAttenVQA. The proposed method achieved an overall accuracy of 64.76%, significantly outperforming the LSTM+CNN baseline (46.40%).

Crucially, on "hard" questions requiring deep reasoning (e.g., determining player locations on the field), the graph-based model achieved 69.63% accuracy, while neural baselines struggled at approximately 15%. The Bayesian inference module proved exceptionally robust, identifying the "player" role with 99% accuracy.

Conclusion

By moving away from unstructured end-to-end models toward a graph-based reasoning framework, this research demonstrates that interpretability and structured logic can significantly improve VQA performance in complex domains. Future work will focus on expanding the system to handle open-ended natural language and integrating broader external knowledge graphs.

Summary updated

Abstract

Visual Query Answering (VQA) is of great significance in offering people convenience: one can raise a question for details of objects, or high-level understanding about the scene, over an image. This paper proposes a novel method to address the VQA problem. In contrast to prior works, our method that targets single scene VQA, replies on graph-based techniques and involves reasoning. In a nutshell, our approach is centered on three graphs. The first graph, referred to as inference graph GI , is constructed via learning over labeled data. The other two graphs, referred to as query graph Q and entity-attribute graph GEA, are generated from natural language query Qnl and image Img, that are issued from users, respectively. As GEA often does not take sufficient information to answer Q, we develop techniques to infer missing information of GEA with GI . Based on GEA and Q, we provide techniques to find matches of Q in GEA, as the answer of Qnl in Img. Unlike commonly used VQA methods that are based on end-to-end neural networks, our graph-based method shows well-designed reasoning capability, and thus is highly interpretable. We also create a dataset on soccer match (Soccer-VQA) with rich annotations. The experimental results show that our approach outperforms the state-of-the-art method and has high potential for future investigation.
BibTeX
@inproceedings{xiong2019visual,
  author = {Peixi Xiong and Huayi Zhan and Xin Wang and Baivab Sinha and Ying Wu},
  title = {{Visual Query Answering by Entity-Attribute Graph Matching and Reasoning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  doi = {10.1109/cvpr.2019.00855},
}
Figure