Abstract
A compositional question refers to a question that contains multiple visual concepts (e.g., objects, attributes, and relationships) and requires compositional reasoning to answer. Existing VQA models can answer a compositional question well, but cannot work well in terms of reasoning consistency in answering the compositional question and its sub-questions. For example, a compositional question for an image is: “Are there any elephants to the right of the white bird?” and one of its sub-questions is “Is any bird visible in the scene?”. The models may answer “yes” to the compositional question, but “no” to the sub-question. This paper presents a dialog-like reasoning method for maintaining reasoning consistency in answering a compositional question and its sub-questions. Our method integrates the reasoning processes for the sub-questions into the reasoning process for the compositional question like a dialog task, and uses a consistency constraint to penalize inconsistent answer predictions. In order to enable quantitative evaluation of reasoning consistency, we construct a GQA-Sub dataset based on the well-organized GQA dataset. Experimental results on the GQA dataset and the GQA-Sub dataset demonstrate the effectiveness of our method.
Introduction: The Problem of Reasoning Inconsistency
Compositional Visual Question Answering (VQA) involves answering complex queries that require multiple logical steps, such as "Are there any elephants to the right of the white bird?" To solve this, a model must successfully identify the bird, verify its color, locate the spatial region to its right, and finally detect elephants within that region.
While state-of-the-art models have become increasingly accurate at answering these complex questions, they often suffer from a fundamental flaw: Reasoning Inconsistency. A model might correctly answer "Yes" to a complex query while failing on the foundational sub-questions that compose it. This suggests that models are often relying on statistical shortcuts or dataset biases rather than genuine multi-step reasoning.
As shown in the examples above, models frequently contradict themselves. In the first case, the model claims there are elephants near a bird while simultaneously claiming no bird is visible. This paper aims to bridge this gap by forcing models to be "self-consistent" through a dialog-like reasoning framework.
Proposed Methodology
The authors propose a method that treats the reasoning process as a structured, iterative conversation between the model and the visual scene.
1. Visual Graph Representation
The image is first represented as a graph where nodes correspond to objects (encoded with appearance and location features) and edges represent the relationships between them. This provides a structured foundation for relational reasoning.
2. Language-Guided Graph Convolution
The model performs iterative updates to the graph nodes using language-guided graph convolutions. The number of reasoning iterations is dynamically determined by the complexity of the question, specifically the number of nouns identified via Part-of-Speech (POS) tagging.
3. Dialog-like Integration
Rather than treating sub-questions as separate tasks, the model integrates them into the main question's reasoning path. As the model "thinks" through the main question, it stops at intermediate steps to answer relevant sub-questions. This ensures that the internal state used to answer a sub-question is the same state used to build toward the final answer.
4. Consistency Constraint ($L_{cons}$)
The core innovation is a novel loss function that penalizes the model if it is more confident in the answer to a complex question than it is in the answer to a foundational sub-question. Logically, if a model is uncertain about the existence of a bird, it should not be certain about the position of elephants relative to that bird. This constraint forces the model to build a solid evidentiary chain.
The GQA-Sub Dataset
To evaluate this approach, the authors introduced GQA-Sub, a benchmark created by automatically decomposing GQA questions into their constituent sub-questions using ground-truth scene graphs and functional programs. The dataset contains over 350,000 sub-questions for training and 45,000 for validation, providing a rigorous testbed for measuring logical consistency.
Experimental Results
The proposed model outperforms standard holistic models (like LCGN) and modular models (like MMN), particularly in consistency metrics.
- Accuracy: On the GQA validation set, the model achieved 66.26%, surpassing the MMN baseline of 65.05%.
- Reasoning Consistency (RC): The authors proposed the RC(k) metric, which measures the percentage of correctly answered main questions where at least k sub-questions were also answered correctly. The improvement here was dramatic: while standard models like LCGN scored ~57.37% on RC(1), the proposed method reached 71.47%.
These results indicate that while data augmentation (simply adding sub-questions to training) helps, the specific dialog-like architecture and consistency loss are essential for achieving robust, logical reasoning.
Conclusion and Future Work
This research highlights that accuracy alone is an insufficient metric for VQA; models must also demonstrate reasoning consistency. By integrating sub-question reasoning into a unified dialog-like framework and applying a confidence-based consistency constraint, the authors have developed a model that is not only more accurate but significantly more logical.
Future work may involve replacing heuristic iteration counts with reinforcement learning-based policy networks to dynamically decide when enough information has been gathered to answer specific sub-questions, further moving toward truly autonomous visual reasoning.
Summary updated