Looking Beyond the Pixels: Evaluating Visual Metaphor Understanding in VLMs

Manishit Kundu, Sumit Shekhar, Pushpak BhattacharyyaEMNLP 2025

TL;DR

This paper introduces ImageMet, a novel dataset for evaluating visual metaphor understanding in Vision-Language Models (VLMs). They benchmark current VLMs on two tasks, Visual Metaphor Captioning (VMC) and Visual Metaphor VQA (VM-VQA), revealing that while models excel at surface-level associations, they still lag significantly behind humans in genuine conceptual reasoning for visual metaphors.

What Problem Does This Paper Address?

Visual metaphors are a fundamental aspect of human communication, allowing us to represent abstract concepts through concrete imagery. Understanding them requires both perceptual reasoning (interpreting visual cues) and conceptual reasoning (mapping source and target domains). While VLMs have made strides in image captioning and visual question answering, their ability to truly grasp the nuanced, often subjective, meaning of visual metaphors remains largely unexplored. Existing benchmarks are limited, and current VLMs often rely on learned associations rather than deep understanding, making it difficult to assess their true capabilities in this area. This paper aims to bridge this gap by providing a dedicated dataset and evaluation framework for visual metaphor understanding.

Methods

The authors introduce two primary tasks for evaluation:

  • Visual Metaphor Captioning (VMC): Models generate a caption that verbalizes the intended meaning of a visual metaphor. They explore both "Open Format" (free-form captions) and "Simile Format" (strict "A is as B as C" structure).
  • Visual Metaphor VQA (VM-VQA): Models identify one of the three metaphor components (Primary, Secondary, or Attribute) from multiple-choice options.

To improve VLM performance, they propose a Chain-of-Thought (CoT) prompting technique. This multi-stage approach involves:

  1. An initial model guess based on the image.
  2. Scene graph generation to capture object relationships.
  3. A structured CoT process to guide the model from objective observations to abstract inferences about source, target, and attribute.
  4. Refinement of the caption based on the analysis.

They also fine-tune existing VLMs on their newly created dataset.

Datasets

The core contribution is the introduction of ImageMet:

  • ImageMet: A dataset comprising 2177 synthetically generated image-simile pairs for training and validation, and 350 manually curated images with simile-style captions for the test set. The synthetic data is generated through an automated pipeline involving noun selection, adjective association, and image generation using Stable Diffusion. The manual test set aims to minimize ambiguity and ensure challenging evaluation.
  • VFLUTE: Used for comparison and fine-tuning, though noted to contain more incidental metaphors and a smaller, less diverse test set.

Key Insights and Ideas

  • VLMs struggle with genuine conceptual reasoning: Despite strong performance on automatic metrics, models often rely on surface-level associations and learned patterns rather than deep understanding of metaphorical mappings.
  • Automatic metrics are insufficient: Metrics like SBERT-Similarity and BLEU correlate only moderately with human judgment for metaphor captioning, highlighting the need for more holistic evaluation.
  • Simile format aids understanding: Fine-tuning models on the structured "A is as B as C" simile format significantly improves their ability to understand and verbalize metaphor components.
  • CoT prompting offers a significant boost: The proposed Chain-of-Thought reasoning technique consistently enhances performance across models, guiding them towards more accurate metaphorical interpretations.
  • Human evaluation is crucial: Human assessments reveal that models still lag behind humans in semantic consistency, indicating a gap in compositional understanding and abstract reasoning.

Key Results

  • Dataset Impact: Fine-tuning VLMs on ImageMet yields substantial performance gains, with improvements of +4.67% in VMC (SBERT-Similarity) and +9.3% in VM-VQA (Accuracy) on average.
  • Prompting Effectiveness: The task-specific CoT prompting strategy outperforms standard few-shot baselines by +1.99% in VMC and +5.21% in VM-VQA.
  • Model Performance: GPT-40 with MS+CoT achieves the strongest baseline performance. Among open-source models, Qwen2-IM (fine-tuned on ImageMet) shows significant improvements.
  • Human vs. Model: While models can outperform humans on automatic metrics for VMC, human evaluations show superior semantic consistency, indicating a deeper understanding. For VM-VQA, humans achieve near-perfect accuracy (93% for Primary, 92% for Secondary, 90% for Attribute), far exceeding models.

Limitations and Future Work

  • Automated Evaluation Shortcomings: The authors acknowledge that current automatic metrics may not fully capture the nuance, creativity, or interpretive depth of visual metaphors, leading to moderate correlations with human judgments. Future work could focus on developing more robust automated evaluation protocols.
  • Resource Constraints: The study was limited to models with parameters up to 11B. Investigating larger open-source models (over 20B) is suggested.
  • Dataset Scalability vs. Diversity: While ImageMet is scalable due to its automated pipeline, the diversity of generated metaphors might be constrained by the generation capabilities.
  • Model Limitations: Models still struggle with domain swapping, visual representation bias (conflating literal and metaphorical meanings), and implicit metaphors where cues are subtle.
  • Future Work: The authors suggest further research into developing models capable of more nuanced perceptual and conceptual reasoning for visual metaphors, and exploring more robust evaluation methods.

Summary updated

BibTeX
@inproceedings{kundu2025looking,
  author = {Manishit Kundu and Sumit Shekhar and Pushpak Bhattacharyya},
  title = {{Looking Beyond the Pixels: Evaluating Visual Metaphor Understanding in VLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}

Loading related content...

Figure