TL;DR
This paper introduces VariNLI-VE, a new dataset designed to study how human disagreement shifts when moving from text-only Natural Language Inference (NLI) to image-based Visual Entailment (VE). Contrary to the intuition that "a picture is worth a thousand words" and should resolve ambiguity, the authors find that visual evidence often restructures rather than reduces disagreement, while current Vision-Language Models (VLMs) struggle significantly more with visual premises than textual ones.
What Problem Does This Paper Address?
In traditional machine learning, we often treat human disagreement as "noise" to be averaged away into a single ground-truth label. However, recent research into Human Label Variation (HLV) suggests that this disagreement is actually a valuable signal reflecting valid, differing interpretations of ambiguous input.
While HLV has been studied in text-only tasks like NLI (e.g., "Does sentence A imply sentence B?"), we know much less about how it behaves in multimodal settings like Visual Entailment (where the premise is an image). One might assume that an image provides more concrete evidence than a short caption, thereby reducing human disagreement. This paper investigates whether that assumption holds true and evaluates how well modern VLMs capture these human nuances.
Methods
The authors constructed the VariNLI-VE dataset by selecting 225 instances from the SNLI dataset that were originally labeled as "neutral." They then performed a controlled re-annotation process:
- Paired Annotation: The same five annotators evaluated both the textual version (NLI) and the visual version (VE) of each instance.
- Joint Collection: For every instance, annotators provided a label (Entailment, Neutral, or Contradiction), a confidence score (1–6), and a natural language explanation.
- Model Evaluation: The researchers tested 13 state-of-the-art VLMs (including Gemma 3, InternVL3.5, and Qwen3-VL) using zero-shot prompting to see if their predictive distributions aligned with human label distributions.
Datasets
- VariNLI-VE (New): Contains 2,250 annotations across 225 paired NLI-VE instances. It is unique because it provides multiple annotations and explanations from the same individuals for both modalities.
- SNLI / e-SNLI-VE: The source datasets from which the premises and hypotheses were drawn.
Key Insights and Ideas
- Disagreement is Restructured, Not Reduced: Adding an image doesn't necessarily make the "correct" answer obvious. While disagreement decreased for 63 instances, it actually increased for 85 others. Visual evidence introduces new details that different people interpret in different ways.
- Rationale Shifts: One of the most striking findings is that annotators often keep the same label but change their reasoning when shown the image. For example, an annotator might label a case as "Contradiction" in both NLI and VE, but cite a textual detail in the first and a specific visual occlusion in the second.
- The VLM "Modality Gap": Even the strongest models perform significantly worse on VE than NLI. This suggests that while VLMs are becoming excellent at processing text, their ability to perform stable logical inference grounded in visual evidence still lags behind.
- Confidence vs. Agreement: Human confidence remains high even when they disagree with each other. This suggests that HLV isn't just about people being "unsure"; it's about people being certain of different, competing interpretations.
Key Results
- Label Transitions: 51.1% of instances changed their majority label when moving from text to image. As seen in Figure 2, many originally "Neutral" text cases shifted toward "Contradiction" or "Entailment" once the visual evidence was available.
- Model Performance Drop: Accuracy consistently dropped from NLI to VE across all model families. For instance, the InternVL3.5-4B model achieved ~91% accuracy on NLI but plummeted to ~55% on VE.
- Contradiction Blindness: VLMs are particularly bad at identifying visual evidence that contradicts a hypothesis. They tend to over-predict "Entailment" or "Neutral" labels in VE settings.
- Uncertainty Alignment: Model uncertainty (measured by entropy) only weakly correlates with human uncertainty. Models are often "confident" in their errors, especially in the visual modality.
Limitations and Future Work
The authors acknowledge that the dataset scale (225 instances) is relatively small due to the high cost of collecting quality human explanations. Additionally, the study focused specifically on "Neutral" instances from SNLI, as these are known to be the most sensitive to modality shifts.
Future work could expand this to a larger variety of initial labels and explore whether the observed "modality gap" in VLMs is due to textual shortcuts learned during training or a fundamental difficulty in visual reasoning. The authors also suggest that future VLM evaluation should move beyond majority-vote accuracy and toward matching the full distribution of human interpretations.
Summary updated