Abstract
Visual reasoning tasks such as visual question answering (VQA) require an interplay of visual perception with reasoning about the question semantics grounded in perception. However, recent advances in this area are still primarily driven by perception improvements rather than reasoning. Neuro-symbolic models bring the benefits of compositional reasoning to VQA, but they are still entangled with visual representation learning, making neural reasoning hard to improve and assess independently. To address this, we propose (1) a framework to isolate and evaluate the reasoning aspect of VQA separately from its perception, and (2) a novel top-down calibration technique that allows the model to answer reasoning questions even with imperfect perception. We introduce a differentiable first-order logic (FOL) formalism for VQA that explicitly decouples question answering from visual perception.
Introduction
In Visual Question Answering (VQA), high accuracy can often be deceptive. It is frequently unclear whether a model correctly answers a complex question like "Is the man wearing a red hat?" because it logically processed the relationships between entities or because its object detector simply excelled at identifying "red hats." Most contemporary VQA models "entangle" perception and reasoning by processing images and text into a shared latent space. This entanglement makes it nearly impossible to discern if a model is truly reasoning or merely exploiting statistical biases.
While datasets like GQA have moved toward multi-step reasoning, they still primarily evaluate perception. Research indicates that a model with perfect "ground-truth" vision can achieve 96% accuracy on GQA, suggesting that the reasoning component is often trivial if the visual input is flawless. To move forward, we must decouple these two components.
The $\nabla$-FOL Framework
The core of our approach is $\nabla$-FOL (Differentiable First-Order Logic), a neuro-symbolic framework that translates natural language questions into formal logic expressions. Unlike traditional symbolic logic, $\nabla$-FOL is fully differentiable, allowing it to be integrated into neural networks.
The framework utilizes "Neural Visual Oracles" to predict the likelihood of various predicates, such as is_red(object_i) or is_left_of(obj_i, obj_j). These probabilities are then propagated through logical operators—AND, OR, NOT, EXISTS, FORALL—using a dynamic programming strategy. This ensures that the computation remains efficient (polynomial time) while maintaining the expressive power of first-order logic.
Top-Down Contextual Calibration
Pure logic can be brittle when faced with noisy visual inputs. For instance, if an object detector misidentifies a "husky" as a "wolf," a strict logical engine will fail the question "Is there a husky in the room?" To mitigate this, we propose Top-Down Contextual Calibration.
This technique employs a Bi-LSTM "calibrator" that analyzes the question context to "nudge" visual attention. If a question mentions a "broken wooden chair," the calibrator can excite the model's attention toward objects that might be chairs, even if the raw visual signal for "broken" is weak. This is achieved through various attention transformation functions.
These functions act as a "soft threshold." While the calibrator won't invent an object from zero signal, it can confirm "fuzzy" signals using linguistic context, effectively allowing the model to "fill in the holes" of imperfect perception using learned world knowledge.
Experimental Evaluation and Insights
We evaluated our framework on the GQA (Genome Question Answering) dataset, specifically the "balanced" Test-Dev split to minimize language bias. Our experiments yielded several key insights:
- The Perception Bottleneck: A baseline version of $\nabla$-FOL (pure logic without calibration) achieves approximately 52% accuracy using standard visual features, establishing a baseline for what is solvable by perception alone.
- Disentangled Evaluation: By splitting the dataset into "Easy" (solvable by the base model) and "Hard" (where perception fails) subsets, we created a new metric to evaluate the reasoning power of black-box models like MAC and LXMERT.
- Model Comparison: On the "Hard" subset, LXMERT achieved 38.43% accuracy compared to MAC’s 30.54%. This suggests that LXMERT’s massive pre-training on vision-language data provides superior cross-modal robustness and reasoning capability.
- Calibration Impact: Adding the top-down calibrator to $\nabla$-FOL dramatically improved accuracy on "Hard" binary questions from 4.42% to 37.82%, demonstrating the power of linguistic context in correcting visual errors.
Conclusion
The $\nabla$-FOL framework provides a robust method for disentangling perception from reasoning in VQA. By treating logic as a differentiable program and incorporating top-down calibration, we can both pinpoint model failures and improve performance on noisy data. While current reliance on a semantic parser is a limitation for "in-the-wild" language, future work will focus on integrating logical awareness directly into the pre-training of large-scale transformers.
Summary updated