Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization

Yue Zhang, Liqiang Jing, Vibhav Gogate

Introduction

In the field of computer vision and natural language processing, Visual Entailment (VE) has traditionally been treated as a static classification problem. Given an image (premise) and a sentence (hypothesis), models are tasked with determining if the relationship is one of entailment, contradiction, or neutrality. However, human reasoning is inherently non-monotonic; new information can frequently strengthen or weaken existing conclusions. This paper introduces Defeasible Visual Entailment (DVE), a novel task and benchmark designed to evaluate how Large Vision-Language Models (LVLMs) handle the dynamic nature of evidence.

The Defeasible Visual Entailment Task

Unlike standard VE, DVE focuses on the "delta" in reasoning. The task involves an image premise $P$, a textual hypothesis $H$, and a textual update $U$. The goal is to determine how $U$ affects the entailment relationship between $P$ and $H$.

Figure 1: An overview of the DVE task. Given an image and a hypothesis, an update can either strengthen or weaken the logical link.

The authors define two primary sub-tasks:

  1. Classification: Identifying whether a given update $U$ is a strengthener (making $H$ more likely given $P$) or a weakener (making $H$ less likely).
  2. Generation: Producing a textual update $U$ that achieves a specific goal (e.g., "write a weakener for this hypothesis").

Benchmark Dataset Construction

To facilitate research in this area, the authors curated a comprehensive DVE Benchmark by integrating three distinct datasets:

  • Flickr30k: Used for high-quality image premises.
  • SNLI (Stanford Natural Language Inference): Used to derive textual hypotheses.
  • $\delta$-NLI: Provided human-annotated "updates" that logically shift entailment.

The resulting dataset contains over 93,000 training samples, providing a rich ground for training and evaluating models on complex multimodal logic.

Methodology: Inference-Aware Evaluation and Optimization

A significant challenge in DVE is that traditional NLP metrics like BLEU, ROUGE, or METEOR fail to capture logical impact, often showing negative correlation with human judgment. To address this, the researchers developed a three-pronged approach.

1. Inference-Aware Evaluator

The core of the methodology is a specialized evaluator designed to measure entailment strength. As shown in the architecture below, it utilizes a ResNet-50 for visual feature extraction and BERT for textual encoding.

Figure 6: The architecture of the proposed DVE framework, featuring multimodal embeddings, feature fusion, and multitask learning.

The model is trained using pairwise contrastive learning. It learns to assign higher scores to (Premise, Hypothesis + Strengthener) pairs than to (Premise, Hypothesis + Neutral) pairs, and lower scores to (Premise, Hypothesis + Weakener) pairs. This allows the evaluator to act as a continuous reward signal for logical consistency.

2. Reward-Driven Optimization

To improve the generation capabilities of LVLMs, the authors implemented a "Critique-Refine" loop. An LVLM (such as GPT-4o) generates an initial update. This update is then scored by the Inference-Aware Evaluator. If the update fails to sufficiently shift the entailment score in the desired direction, the model receives the score as feedback and is prompted to refine the text. This iterative process forces models to move beyond simple image description and engage in actual argumentation.

Experimental Results

The researchers' evaluation revealed several key insights:

  • Metric Reliability: The proposed Inference-Aware Evaluator achieved a high correlation with human judgment ($r \approx 0.82$), whereas traditional metrics and even CLIPScore ($r \approx 0.10$) performed poorly.
  • Classification Performance: GPT-4o demonstrated strong zero-shot reasoning with 81.76% accuracy. Interestingly, smaller fine-tuned models like CLIP (71.1%) outperformed larger general-purpose models like LLaVA-1.5 (52.07%), suggesting that specialized training is crucial for this task.
  • Generation Gains: The reward-driven optimization significantly improved the effectiveness of generated updates. For GPT-4o, the "strengthening" score improved from 3.64 to 4.06, and the "weakening" score became more pronounced, moving from -2.52 to -2.92.

Conclusion

Defeasible Visual Entailment represents a significant step toward more robust and human-like AI reasoning. By moving from categorical labels to a continuous spectrum of entailment strength, the DVE framework provides a more nuanced way to evaluate and optimize multimodal models. While a performance gap remains between current state-of-the-art models and human reasoning, the combination of specialized benchmarks, inference-aware evaluators, and reward-driven optimization offers a promising path forward for the next generation of LVLMs.

Summary updated

Abstract

We introduce a new task called Defeasible Visual Entailment (DVE), where the goal is to allow the modification of the entailment relationship between an image premise and a text hypothesis based on an additional update. While this concept is well-established in Natural Language Inference, it remains unexplored in visual entailment. At a high level, DVE enables models to refine their initial interpretations, leading to improved accuracy and reliability in various applications such as detecting misleading information in images, enhancing visual question answering, and refining decision-making processes in autonomous systems. Existing metrics do not adequately capture the change in the entailment relationship brought by updates. To address this, we propose a novel inference-aware evaluator designed to capture changes in entailment strength induced by updates, using pairwise contrastive learning and categorical information learning. Additionally, we introduce a reward-driven update optimization method to further enhance the quality of updates generated by multimodal models. Experimental results demonstrate the effectiveness of our proposed evaluator and optimization method.
BibTeX
@inproceedings{zhang2025defeasible,
  author = {Yue Zhang and Liqiang Jing and Vibhav Gogate},
  title = {{Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i24.34792},
}
Figure