Abstract
Traditional Visual Question Answering (VQA) systems often rely on end-to-end neural networks that function as "black boxes," making their reasoning processes difficult to interpret. This paper proposes a novel framework that shifts VQA toward an interpretable, graph-based reasoning approach. By representing images as Entity-Attribute Graphs (EAG) and utilizing Bayesian Networks to infer implicit properties, the system achieves superior performance on complex, multi-step reasoning tasks. Evaluation on the specialized Soccer-VQA dataset demonstrates that this structured approach significantly outperforms state-of-the-art neural baselines, particularly on high-difficulty queries.
Introduction
Most contemporary VQA models map images and text into a joint mathematical space to predict answers. While effective for simple recognition tasks, these models suffer from two primary limitations: a lack of interpretability and a reliance on dataset statistics rather than genuine logic. They often struggle with "hidden" information—attributes that are not directly visible in pixels but must be inferred from context, such as a player's role or team status in a sports match. This research addresses these flaws by introducing a pipeline that explicitly models entities, attributes, and their relationships.
Methodology
The proposed approach centers on the construction and matching of structured graphs. The process is divided into four distinct stages:
- Visual Processing & Graph Construction: The system detects objects (e.g., players, ball) and extracts explicit attributes like uniform color and field location to build an initial, incomplete Entity-Attribute Graph (EAG).
- Inference Graph (Reasoning): To handle implicit attributes, the authors employ a Bayesian Network trained on domain-specific data. This network infers hidden properties—such as whether a person is a goalkeeper or a referee—based on visible cues like movement direction and position.
- Query Graph Generation: Natural language questions are parsed into structured "Query Graphs." For instance, a query about the number of red players is transformed into a graph searching for entities with the attributes
color: redandrole: player. - Graph Matching: The system performs subgraph isomorphism to find matches for the query graph within the image's EAG, applying arithmetic operations like counting to generate the final answer.
The Soccer-VQA Dataset
To evaluate the framework, the authors introduced Soccer-VQA, a benchmark consisting of 7,900 frames from the 2016 FIFA World Cup. Unlike general-purpose datasets, Soccer-VQA is richly annotated with object locations, roles, actions, and field coordinates. This domain-specific focus requires models to move beyond simple object detection and perform high-level reasoning about game rules and player roles.
Key Results and Insights
The graph-based approach demonstrated a substantial performance advantage over standard neural models like LSTM+CNN and HieCoAttenVQA:
- Overall Accuracy: The proposed method achieved 64.76%, significantly higher than the 49.11% achieved by the best neural baseline.
- Reasoning under Complexity: On the most challenging questions (e.g., identifying players in specific field zones), the model reached 69.63% accuracy, while neural models dropped to approximately 15%.
- Attribute Inference: The Bayesian module proved highly effective, identifying "players" with 99% accuracy and "goalkeepers" with nearly 90% accuracy.
These results suggest that structured knowledge representations are more robust than joint embeddings for tasks involving counting or complex role identification, where neural networks often fail to distinguish signal from high-dimensional noise.
Conclusion and Future Work
By design, this framework provides full interpretability; a user can inspect the resulting graph to see exactly which entities and attributes led to an answer. While currently optimized for the soccer domain, future work aims to expand this to general-purpose VQA by integrating broader knowledge graphs and handling more diverse, open-ended natural language queries. The integration of external "common sense" knowledge remains a key frontier for handling reasoning tasks that extend beyond the immediate visual evidence.
Summary updated