Logical Implications for Visual Question Answering Consistency

Sergio Tascon-Morales, Pablo Márquez-Neila, Raphael Sznitman

Abstract

Visual Question Answering (VQA) models often exhibit "logical hypocrisy," providing correct answers to complex queries while contradicting themselves on simpler, related ones. We present LI-MOD (Logical-Implication Model), a novel strategy that leverages a language model to infer logical relations between questions and enforces consistency via a specialized loss function. Our method is model-agnostic and significantly improves consistency across general and medical VQA datasets.

Introduction: The Problem of Logical Hypocrisy

Despite advancements in identifying objects and answering visual queries, VQA models often lack a fundamental grasp of logic. A model might correctly state that "there is snow on the ground" but simultaneously claim it is "the middle of summer." This discrepancy suggests that models rely on statistical pattern matching rather than genuine reasoning.

Conventional VQA models often produce inconsistent answers for logically related questions, whereas the proposed method uses inferred relations to maintain consistency.

Existing solutions often require extensive human-annotated data or rely on narrow assumptions about question relationships. The authors propose an automated, generalizable method to enforce logical consistency across any pair of related questions without requiring manual mapping.

Methodology: The LI-MOD Framework

The proposed LI-MOD approach consists of two primary components:

  1. Logical Relation Inference: A BERT-based language model is fine-tuned on Natural Language Inference (NLI) tasks. It evaluates pairs of question-answer propositions (e.g., "Is it snowing? Yes" and "Is it winter? Yes") to determine if one implies the other, if they are equivalent, or if they are unrelated.
  2. Consistency Loss: A novel loss function is introduced to penalize logical contradictions. If a model is confident in a "sufficient" condition but denies its "necessary" condition, the consistency loss applies a heavy penalty.

This framework is model-agnostic, allowing it to be integrated into existing VQA architectures like BAN or LXMERT without altering their underlying neural structures.

Experimental Results

The method was evaluated on the VQA Introspect dataset and the DME Dataset for medical imaging.

Qualitative results comparing LI-MOD (Ours) against baselines. Red sirens indicate logical inconsistencies in the model outputs.

Key Findings

  • General VQA: Using the LXMERT backbone on the Introspect dataset, LI-MOD achieved 75.17% accuracy and 78.75% consistency, outperforming standard baselines.
  • Medical VQA: In the high-stakes DME dataset, the model reached 95.78% consistency, a significant jump from the 89.95% baseline.
  • Training vs. Post-hoc Fixes: The authors demonstrated that training for consistency is superior to simply "flipping" inconsistent answers at test time, which often degrades overall accuracy.

Conclusion and Future Work

LI-MOD successfully addresses the "logical hypocrite" problem by treating question-answer pairs as logical propositions. While currently optimized for binary questions, future work aims to extend this logic to open-ended queries and reduce dependency on the accuracy of the underlying NLI model. The development of a broader database of logical relations could further streamline the application of this method to diverse VQA tasks.

Summary updated

Abstract

Despite considerable recent progress in Visual Question Answering (VQA) models, inconsistent or contradictory answers continue to cast doubt on their true reasoning capabilities. However, most proposed methods use indirect strategies or strong assumptions on pairs of questions and answers to enforce model consistency. Instead, we propose a novel strategy intended to improve model performance by directly reducing logical inconsistencies. To do this, we introduce a new consistency loss term that can be used by a wide range of the VQA models and which relies on knowing the logical relation between pairs of questions and answers. While such information is typically not available in VQA datasets, we propose to infer these logical relations using a dedicated language model and use these in our proposed consistency loss function. We conduct extensive experiments on the VQA Introspect and DME datasets and show that our method brings improvements to state-of-the-art VQA models, while being robust across different architectures and settings.
BibTeX
@inproceedings{tasconmorales2023logical,
  author = {Sergio Tascon-Morales and Pablo Márquez-Neila and Raphael Sznitman},
  title = {{Logical Implications for Visual Question Answering Consistency}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00650},
}
Figure