Leveraging Large Models to Evaluate Novel Content: A Case Study on Advertisement Creativity

Zihao Hou, Adriana Kovashka, Xiang Lorraine Li

Presented at EMNLP 2025, this research explores a critical intersection of cognitive science, marketing, and computer vision: Can AI understand what makes an advertisement "creative"? While Large Language Models (LLMs) have shown promise in creative writing, evaluating visual creativity is a significantly more complex task due to its inherent subjectivity and the use of sophisticated visual metaphors.

The Challenge of Subjective Evaluation

In marketing, creativity is a primary driver of consumer behavior, yet it remains notoriously difficult to measure. Unlike standard computer vision tasks with a single "ground truth" (e.g., object detection), creativity is subjective. One viewer might find an ad brilliant, while another finds it mundane or confusing.

Existing AI evaluation frameworks often rely on majority-label prediction, which ignores this diversity of opinion. This paper addresses the gap in visual creativity assessment by moving beyond simple labels to model the distribution of human judgments.

Defining Creativity: Atypicality and Originality

The researchers decomposed creativity into two core dimensions based on established marketing theory:

  1. Atypicality: The degree to which an ad connects usually unrelated objects or ideas.
  2. Originality: The degree to which an ad breaks away from stereotypical thinking.
An example of an atypical advertisement featuring a cow typing on a laptop, used in the study to test visual metaphor recognition.

To test Vision-Language Models (VLMs), the authors proposed three distinct tasks:

  • Distribution Modeling: Prompting the VLM multiple times at a high "temperature" to see if the model's spread of answers matches the spread of human opinions.
  • Disagreement Prediction: Assessing if the model can predict how much humans will disagree on an ad's creativity.
  • Pairwise Preference: Identifying which of two ads humans rated as more creative.

Benchmarking Performance

The study utilized two primary datasets derived from the Pitt Advertisement Dataset: Creative-100 (100 ads with 25 human annotations each) and Atypicality-300 (300 ads focused on the atypicality dimension).

A comparison of human and model score distributions for two different advertisements (A and B), illustrating the distribution modeling task.

Key Insights

  • The Reasoning Bottleneck: Interestingly, smaller models like LLaVA-7B (Mistral-based) often outperformed the larger LLaVA-13B (Vicuna-based) in reasoning-heavy tasks. This suggests that the quality of the underlying language model's reasoning is more critical for evaluating creativity than raw parameter count.
  • Pairwise is "Easy," Absolute is Hard: Models excel at relative comparisons (e.g., "Ad A is better than Ad B") but struggle to assign absolute scores that align with human intuition. GPT-4v led the field here with a pairwise F1 score of 0.97.
  • The "Final Boss" of Subjectivity: Predicting human disagreement remains an unsolved problem. Most models tend to predict "middle-of-the-road" disagreement, failing to identify which specific ads will be polarizing.
  • Visual Metaphor Complexity: VLMs frequently fail on ads requiring "multi-hop" reasoning, such as connecting a specific slang phrase to a visual pun.

Results Summary

  • Correlation: InternVL2-8B showed the strongest alignment with average human ratings, achieving a Spearman correlation of 0.82 for originality.
  • Disagreement Prediction: This was the weakest area for all models; correlations with human standard deviations were near zero, indicating that VLMs cannot yet "sense" the ambiguity that causes human debate.

Limitations and Future Work

The authors note that creativity is culturally dependent; the current human annotators were primarily US-based. Future research could explore "Persona Prompting"—asking VLMs to respond as if they were from a specific demographic—to better simulate diverse human perspectives. By providing a benchmark that respects the subjective nature of art, this work moves us closer to AI that understands the nuance of human expression.

Summary updated

BibTeX
@misc{hou2025leveraging,
  author = {Zihao Hou and Adriana Kovashka and Xiang Lorraine Li},
  title = {{Leveraging Large Models to Evaluate Novel Content: A Case Study on Advertisement Creativity}},
  year = {2025},
  eprint = {2503.00046},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.00046},
}
Figure