Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization

Yue Zhang, Liqiang Jing, Vibhav Gogate

Introduction

Traditional Visual Entailment (VE) is typically treated as a monotonic reasoning task: given an image (premise) and a sentence (hypothesis), a model determines if the relationship is one of entailment, contradiction, or neutrality. However, human reasoning is inherently defeasible; our conclusions are subject to change as new information becomes available. This paper introduces Defeasible Visual Entailment (DVE), a novel task that explores how additional textual "updates" can strengthen or weaken the logical link between an image and a hypothesis.

Example of Defeasible Visual Entailment: updates can strengthen or weaken the entailment relationship between an image and a hypothesis.

Existing AI models often struggle with these nuances. While they might identify basic entailment, they lack the ability to integrate new evidence that undermines or bolsters a conclusion. Furthermore, standard metrics like ROUGE or BLEU are insufficient for evaluating DVE because they focus on word overlap rather than the logical strength of an update.

Methodology and Dataset

The researchers address the DVE challenge through three primary contributions: a large-scale benchmark, a specialized evaluator, and an optimization framework.

The DVE Dataset

To facilitate research, the authors constructed the DVE dataset by integrating three existing resources: Flickr30k (images), SNLI (hypotheses), and $\delta$-NLI (textual updates). By mapping textual premises from $\delta$-NLI to corresponding images in Flickr30k, they created a massive dataset of over 93,000 training samples without the need for expensive manual annotation.

Inference-aware Evaluator

To accurately measure the impact of an update, the authors developed an inference-aware evaluator. This model utilizes a ResNet-50 backbone for visual features and BERT for textual features.

Architecture of the Inference-aware Evaluator, illustrating the fusion of visual and textual features for multitask learning.

The evaluator is trained using pairwise contrastive learning, enabling it to rank updates based on their logical impact. For instance, a "strengthener" should yield a higher entailment score than a neutral caption, which in turn should score higher than a "weakener."

Reward-Driven Optimization

The evaluator also serves as a "critic" in a reward-driven optimization loop. When a Large Vision-Language Model (LVLM) generates an update, the evaluator provides a score as a reward signal. This feedback allows the LVLM to iteratively refine its output to more effectively achieve the intended strengthening or weakening effect.

Key Results and Insights

The study yielded several critical insights into the nature of visual reasoning:

  • Reasoning vs. Word Overlap: Traditional NLP metrics (ROUGE, BLEU) showed a near-zero or even negative correlation with human judgment on DVE tasks. This highlights the necessity of logic-aware evaluation.
  • The Difficulty of Weakening: Models like GPT-4o are naturally more proficient at finding supporting evidence. Generating effective "weakeners" proved to be a significantly more challenging reasoning task, benefiting greatly from the proposed iterative refinement.
  • Human Alignment: The proposed evaluator achieved a high Pearson correlation of 0.8262 with human judgment, vastly outperforming CLIPScore (0.1068).
  • Optimization Performance: The reward-driven method successfully improved GPT-4o's generation. Entailment scores for strengtheners increased from 3.64 to 4.06, while scores for weakeners were successfully pushed lower (from -2.52 to -2.92).

Conclusion

Defeasible Visual Entailment represents a significant step toward more human-like reasoning in AI. By providing a robust benchmark and a logic-sensitive evaluator, this work enables models to better handle the fluid nature of information. Future work will explore applying DVE to real-world scenarios, such as identifying misinformation in social media or improving the reliability of autonomous systems in ambiguous environments.

Summary updated

BibTeX
@inproceedings{zhang2025defeasible,
  author = {Yue Zhang and Liqiang Jing and Vibhav Gogate},
  title = {{Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i24.34792},
}
Figure