Visual Query Answering by Entity-Attribute Graph Matching and Reasoning

Peixi Xiong, Huayi Zhan, Xin Wang, Baivab Sinha, Ying Wu

Abstract

Traditional Visual Question Answering (VQA) systems often rely on end-to-end neural networks that function as "black boxes," making their reasoning processes difficult to interpret. This paper proposes a novel framework that shifts VQA toward an interpretable, graph-based reasoning approach. By representing images as Entity-Attribute Graphs (EAG) and utilizing Bayesian Networks to infer implicit properties, the system achieves superior performance on complex, multi-step reasoning tasks. Evaluation on the specialized Soccer-VQA dataset demonstrates that this structured approach significantly outperforms state-of-the-art neural baselines, particularly on high-difficulty queries.

Introduction

Most contemporary VQA models map images and text into a joint mathematical space to predict answers. While effective for simple recognition tasks, these models suffer from two primary limitations: a lack of interpretability and a reliance on dataset statistics rather than genuine logic. They often struggle with "hidden" information—attributes that are not directly visible in pixels but must be inferred from context, such as a player's role or team status in a sports match. This research addresses these flaws by introducing a pipeline that explicitly models entities, attributes, and their relationships.

Methodology

The proposed approach centers on the construction and matching of structured graphs. The process is divided into four distinct stages:

  1. Visual Processing & Graph Construction: The system detects objects (e.g., players, ball) and extracts explicit attributes like uniform color and field location to build an initial, incomplete Entity-Attribute Graph (EAG).
  2. Inference Graph (Reasoning): To handle implicit attributes, the authors employ a Bayesian Network trained on domain-specific data. This network infers hidden properties—such as whether a person is a goalkeeper or a referee—based on visible cues like movement direction and position.
  3. Query Graph Generation: Natural language questions are parsed into structured "Query Graphs." For instance, a query about the number of red players is transformed into a graph searching for entities with the attributes color: red and role: player.
  4. Graph Matching: The system performs subgraph isomorphism to find matches for the query graph within the image's EAG, applying arithmetic operations like counting to generate the final answer.
Overview of the proposed graph-based VQA approach, illustrating the pipeline from visual analysis to final result.

The Soccer-VQA Dataset

To evaluate the framework, the authors introduced Soccer-VQA, a benchmark consisting of 7,900 frames from the 2016 FIFA World Cup. Unlike general-purpose datasets, Soccer-VQA is richly annotated with object locations, roles, actions, and field coordinates. This domain-specific focus requires models to move beyond simple object detection and perform high-level reasoning about game rules and player roles.

An annotated soccer frame demonstrating the extraction of entities and attributes used to construct the Entity-Attribute Graph.

Key Results and Insights

The graph-based approach demonstrated a substantial performance advantage over standard neural models like LSTM+CNN and HieCoAttenVQA:

  • Overall Accuracy: The proposed method achieved 64.76%, significantly higher than the 49.11% achieved by the best neural baseline.
  • Reasoning under Complexity: On the most challenging questions (e.g., identifying players in specific field zones), the model reached 69.63% accuracy, while neural models dropped to approximately 15%.
  • Attribute Inference: The Bayesian module proved highly effective, identifying "players" with 99% accuracy and "goalkeepers" with nearly 90% accuracy.

These results suggest that structured knowledge representations are more robust than joint embeddings for tasks involving counting or complex role identification, where neural networks often fail to distinguish signal from high-dimensional noise.

Conclusion and Future Work

By design, this framework provides full interpretability; a user can inspect the resulting graph to see exactly which entities and attributes led to an answer. While currently optimized for the soccer domain, future work aims to expand this to general-purpose VQA by integrating broader knowledge graphs and handling more diverse, open-ended natural language queries. The integration of external "common sense" knowledge remains a key frontier for handling reasoning tasks that extend beyond the immediate visual evidence.

Summary updated

Abstract

Visual Query Answering (VQA) is of great significance in offering people convenience: one can raise a question for details of objects, or high-level understanding about the scene, over an image. This paper proposes a novel method to address the VQA problem. In contrast to prior works, our method that targets single scene VQA, replies on graph-based techniques and involves reasoning. In a nutshell, our approach is centered on three graphs. The first graph, referred to as inference graph GI , is constructed via learning over labeled data. The other two graphs, referred to as query graph Q and entity-attribute graph GEA, are generated from natural language query Qnl and image Img, that are issued from users, respectively. As GEA often does not take sufficient information to answer Q, we develop techniques to infer missing information of GEA with GI . Based on GEA and Q, we provide techniques to find matches of Q in GEA, as the answer of Qnl in Img. Unlike commonly used VQA methods that are based on end-to-end neural networks, our graph-based method shows well-designed reasoning capability, and thus is highly interpretable. We also create a dataset on soccer match (Soccer-VQA) with rich annotations. The experimental results show that our approach outperforms the state-of-the-art method and has high potential for future investigation.
BibTeX
@inproceedings{xiong2019visual,
  author = {Peixi Xiong and Huayi Zhan and Xin Wang and Baivab Sinha and Ying Wu},
  title = {{Visual Query Answering by Entity-Attribute Graph Matching and Reasoning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  doi = {10.1109/cvpr.2019.00855},
}
Figure