In the Eye of the Beholder: Human Label Variation in Textual and Visual Entailment

Monica Riedler, Jasmin Orth, Raffaella Bernardi, Barbara PlankarXiv 2026 · arXiv

TL;DR

This paper introduces VariNLI-VE, a new dataset designed to study how human disagreement shifts when moving from text-only Natural Language Inference (NLI) to image-based Visual Entailment (VE). Contrary to the intuition that "a picture is worth a thousand words" and should resolve ambiguity, the authors find that visual evidence often restructures rather than reduces disagreement, while current Vision-Language Models (VLMs) struggle significantly more with visual premises than textual ones.

What Problem Does This Paper Address?

In traditional machine learning, we often treat human disagreement as "noise" to be averaged away into a single ground-truth label. However, recent research into Human Label Variation (HLV) suggests that this disagreement is actually a valuable signal reflecting valid, differing interpretations of ambiguous input.

While HLV has been studied in text-only tasks like NLI (e.g., "Does sentence A imply sentence B?"), we know much less about how it behaves in multimodal settings like Visual Entailment (where the premise is an image). One might assume that an image provides more concrete evidence than a short caption, thereby reducing human disagreement. This paper investigates whether that assumption holds true and evaluates how well modern VLMs capture these human nuances.

Methods

The authors constructed the VariNLI-VE dataset by selecting 225 instances from the SNLI dataset that were originally labeled as "neutral." They then performed a controlled re-annotation process:

  1. Paired Annotation: The same five annotators evaluated both the textual version (NLI) and the visual version (VE) of each instance.
  2. Joint Collection: For every instance, annotators provided a label (Entailment, Neutral, or Contradiction), a confidence score (1–6), and a natural language explanation.
  3. Model Evaluation: The researchers tested 13 state-of-the-art VLMs (including Gemma 3, InternVL3.5, and Qwen3-VL) using zero-shot prompting to see if their predictive distributions aligned with human label distributions.
Figure 1: Setup and analyses. For each VariNLI-VE in- stance, we collect five annotations, including a label (E, N, C), confidence score (1-6), and explanation from the same annotators across NLI and VE tasks, and compare with model predictions and uncertainty estimates.

Datasets

  • VariNLI-VE (New): Contains 2,250 annotations across 225 paired NLI-VE instances. It is unique because it provides multiple annotations and explanations from the same individuals for both modalities.
  • SNLI / e-SNLI-VE: The source datasets from which the premises and hypotheses were drawn.

Key Insights and Ideas

  • Disagreement is Restructured, Not Reduced: Adding an image doesn't necessarily make the "correct" answer obvious. While disagreement decreased for 63 instances, it actually increased for 85 others. Visual evidence introduces new details that different people interpret in different ways.
  • Rationale Shifts: One of the most striking findings is that annotators often keep the same label but change their reasoning when shown the image. For example, an annotator might label a case as "Contradiction" in both NLI and VE, but cite a textual detail in the first and a specific visual occlusion in the second.
  • The VLM "Modality Gap": Even the strongest models perform significantly worse on VE than NLI. This suggests that while VLMs are becoming excellent at processing text, their ability to perform stable logical inference grounded in visual evidence still lags behind.
  • Confidence vs. Agreement: Human confidence remains high even when they disagree with each other. This suggests that HLV isn't just about people being "unsure"; it's about people being certain of different, competing interpretations.

Key Results

  • Label Transitions: 51.1% of instances changed their majority label when moving from text to image. As seen in Figure 2, many originally "Neutral" text cases shifted toward "Contradiction" or "Entailment" once the visual evidence was available.
Figure 2: Label transitions from the original SNLI label to our newly derived NLI and VE majority labels in VariNLI-VE. Ties indicate cases with no majority label among five annotators.
  • Model Performance Drop: Accuracy consistently dropped from NLI to VE across all model families. For instance, the InternVL3.5-4B model achieved ~91% accuracy on NLI but plummeted to ~55% on VE.
  • Contradiction Blindness: VLMs are particularly bad at identifying visual evidence that contradicts a hypothesis. They tend to over-predict "Entailment" or "Neutral" labels in VE settings.
  • Uncertainty Alignment: Model uncertainty (measured by entropy) only weakly correlates with human uncertainty. Models are often "confident" in their errors, especially in the visual modality.
Figure 5: Model performance and uncertainty across NLI and VE. The left half shows accuracy with respect to human majority labels across all instances and on the neutral-only paired instances. The right half shows model uncertainty, measured by normalized predictive entropy and LAC conformal prediction set size, with larger LAC sets indicating higher uncertainty. Accuracy decreases from NLI to VE, while uncertainty generally increases.

Limitations and Future Work

The authors acknowledge that the dataset scale (225 instances) is relatively small due to the high cost of collecting quality human explanations. Additionally, the study focused specifically on "Neutral" instances from SNLI, as these are known to be the most sensitive to modality shifts.

Future work could expand this to a larger variety of initial labels and explore whether the observed "modality gap" in VLMs is due to textual shortcuts learned during training or a fundamental difficulty in visual reasoning. The authors also suggest that future VLM evaluation should move beyond majority-vote accuracy and toward matching the full distribution of human interpretations.

Summary updated

Abstract

Human Label Variation (HLV) is increasingly recognised as an informative signal that reflects meaningful differences in how people interpret the same input. While HLV has been studied extensively in Natural Language Inference (NLI), much less is known about HLV in Visual Entailment (VE; Xie et al., 2019), where the textual premise is replaced by an image, keeping the hypothesis and label space fixed. We introduce VariNLI-VE, a dataset for studying how HLV changes between paired NLI and VE instances. We re-annotate a subset of originally neutral SNLI (Bowman et al., 2015) instances and their visual counterparts, with the same annotators evaluating both the textual and visual versions. Each annotation includes a label, confidence score, and explanation, yielding 2,250 annotations. Labels and explanations are collected jointly, providing ecologically valid explanations. Because the image may reveal information not stated in the text, one might expect disagreement to decrease in VE. Instead, we find no clear overall reduction: disagreement decreases for some instances while it increases for others. Explanations reveal further variation, as annotators can retain the same label while changing their rationale. When evaluating vision-language models (VLMs) on the paired data, we find lower accuracy, weaker alignment with human label distributions, and generally higher uncertainty in VE than in NLI.
BibTeX
@misc{riedler2026eye,
  author = {Monica Riedler and Jasmin Orth and Raffaella Bernardi and Barbara Plank},
  title = {{In the Eye of the Beholder: Human Label Variation in Textual and Visual Entailment}},
  year = {2026},
  eprint = {2611.17954},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17954},
}
Figure