AbsVis – Benchmarking How Humans and Vision-Language Models “See” Abstract Concepts in Images

Tarun Tater, Diego Frassinelli, Sabine Schulte im WaldeEMNLP 2025

TL;DR

This paper introduces AbsVis, a new dataset and benchmark for understanding how humans and AI models (Vision-Language Models or VLMs) interpret abstract concepts in images. They found that while VLMs are improving, they still struggle with the nuances of abstract thought compared to humans, and that explanations play a crucial role in bridging this gap.

What Problem Does This Paper Address?

We often use abstract concepts like "peace," "freedom," or "justice" to describe the world around us, but these ideas are notoriously difficult to represent visually. Unlike concrete objects (like a "car" or "tree"), abstract concepts lack direct visual anchors. This makes it challenging to study how humans understand them in images and, crucially, how well AI models can grasp them. Current AI models, particularly Vision-Language Models (VLMs), excel at recognizing objects but often falter when it comes to interpreting the deeper, more subjective meanings an image might convey. Existing datasets are often too small or too focused on concrete objects, leaving a significant gap in our understanding of AI's ability to "see" abstract concepts.

Methods

The researchers created AbsVis, a novel dataset designed to tackle this problem. It consists of:

  1. Concept-Explanation Pairs: Over 10,000 pairs where humans and two leading VLMs (Qwen and LLaVA) provide abstract concepts for 675 images, along with textual explanations of why they chose that concept. This allows for a direct comparison of how humans and AI attribute and justify abstract concepts.
  2. Human Preference Judgments: Nearly 3,000 judgments where humans indicate which concept-explanation pairs they find most representative of an image. This helps understand what makes an attribution "good" from a human perspective.
  3. DPO Fine-Tuning: They explored using Direct Preference Optimization (DPO) to fine-tune VLMs, using human preferences as a guide to improve their alignment with human interpretations of abstract concepts.

Datasets

The primary contribution is the AbsVis dataset, which includes:

  • 675 images.
  • 10,125 human-written concept-explanation pairs.
  • 4,050 VLM-generated concept-explanation pairs (from Qwen and LLaVA).
  • 2,680 human preference judgments on concept-explanation pairs.

The images were sourced from the YFCC100M dataset. For concept abstractness ratings, they leveraged the Brysbaert norms.

Key Insights and Ideas

  • Human Diversity is High: Humans themselves don't always agree on what abstract concepts an image evokes, showing the inherent subjectivity.
  • VLMs are More Consistent (Sometimes Too Much): VLMs tend to produce more consistent attributions than humans, but this can sometimes mean they miss the richer, more varied interpretations humans offer.
  • Explanations Matter: While VLMs might struggle with abstract concepts, their explanations can significantly improve how humans perceive the attribution's quality, making even less common concepts more relatable.
  • Overlapping Concepts are King: Concept-explanation pairs that are attributed by both humans and VLMs are consistently preferred by humans, suggesting these represent a sweet spot of recognizable yet nuanced interpretations.
  • VLMs as Judges: VLMs can act as reasonable proxies for human preferences, especially for explanations, opening doors for more scalable evaluation.

Key Results

  • Concept Attribution: Humans attribute a wider variety of concepts than VLMs. VLMs tend to produce longer explanations than humans.
  • Preference: Overlapping concepts (attributed by both humans and VLMs) are most preferred. VLM-exclusive concepts were preferred more often than human-exclusive ones, suggesting VLMs can generate highly representative concepts.
  • Explanation Impact: Adding explanations significantly improved preferences for overlapping concepts. VLM explanations were often preferred over human ones, possibly due to their length and structure.
  • VLM Alignment: VLMs showed moderate to strong correlations with human preferences, especially for explanations.
  • DPO Fine-Tuning: Fine-tuning VLMs using DPO with preference data led to consistent improvements in their ability to align with human judgments.

Limitations and Future Work

  • AI Assistance: While efforts were made to filter out AI-generated human responses, the possibility of some annotators using AI assistants cannot be entirely ruled out.
  • VLM Sensitivity: VLM outputs can be sensitive to prompt variations, potentially affecting consistency.
  • Limited Scope: The preference evaluations were limited to concepts with Brysbaert ratings, and the study focused on only two VLMs (Qwen and LLaVA).
  • Explanation Formatting: Variations in explanation formatting might have influenced human preferences.
  • Future Work: The authors suggest exploring a broader set of VLMs, investigating how to capture broader conceptual consensus versus individual interpretations, and exploring other fine-tuning approaches beyond DPO.

Summary updated

BibTeX
@inproceedings{tater2025absvis,
  author = {Tarun Tater and Diego Frassinelli and Sabine Schulte im Walde},
  title = {{AbsVis – Benchmarking How Humans and Vision-Language Models “See” Abstract Concepts in Images}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure