Abstract
Visual Question Answering (VQA) models often exhibit "logical hypocrisy," providing correct answers to complex queries while contradicting themselves on simpler, related ones. We present LI-MOD (Logical-Implication Model), a novel strategy that leverages a language model to infer logical relations between questions and enforces consistency via a specialized loss function. Our method is model-agnostic and significantly improves consistency across general and medical VQA datasets.
Introduction: The Problem of Logical Hypocrisy
Despite advancements in identifying objects and answering visual queries, VQA models often lack a fundamental grasp of logic. A model might correctly state that "there is snow on the ground" but simultaneously claim it is "the middle of summer." This discrepancy suggests that models rely on statistical pattern matching rather than genuine reasoning.
Existing solutions often require extensive human-annotated data or rely on narrow assumptions about question relationships. The authors propose an automated, generalizable method to enforce logical consistency across any pair of related questions without requiring manual mapping.
Methodology: The LI-MOD Framework
The proposed LI-MOD approach consists of two primary components:
- Logical Relation Inference: A BERT-based language model is fine-tuned on Natural Language Inference (NLI) tasks. It evaluates pairs of question-answer propositions (e.g., "Is it snowing? Yes" and "Is it winter? Yes") to determine if one implies the other, if they are equivalent, or if they are unrelated.
- Consistency Loss: A novel loss function is introduced to penalize logical contradictions. If a model is confident in a "sufficient" condition but denies its "necessary" condition, the consistency loss applies a heavy penalty.
This framework is model-agnostic, allowing it to be integrated into existing VQA architectures like BAN or LXMERT without altering their underlying neural structures.
Experimental Results
The method was evaluated on the VQA Introspect dataset and the DME Dataset for medical imaging.
Key Findings
- General VQA: Using the LXMERT backbone on the Introspect dataset, LI-MOD achieved 75.17% accuracy and 78.75% consistency, outperforming standard baselines.
- Medical VQA: In the high-stakes DME dataset, the model reached 95.78% consistency, a significant jump from the 89.95% baseline.
- Training vs. Post-hoc Fixes: The authors demonstrated that training for consistency is superior to simply "flipping" inconsistent answers at test time, which often degrades overall accuracy.
Conclusion and Future Work
LI-MOD successfully addresses the "logical hypocrite" problem by treating question-answer pairs as logical propositions. While currently optimized for binary questions, future work aims to extend this logic to open-ended queries and reduce dependency on the accuracy of the underlying NLI model. The development of a broader database of logical relations could further streamline the application of this method to diverse VQA tasks.
Summary updated