Leveraging Large Models for Evaluating Novel Content: A Case Study on Advertisement Creativity

Zhaoyi Joey Hou, Adriana Kovashka, Xiang Lorraine Li

Introduction

Evaluating creativity is a complex challenge due to its inherent subjectivity. In marketing, a "creative" advertisement is defined not just by aesthetic appeal but by its ability to break established norms and capture consumer attention. Traditional AI evaluation frameworks typically rely on "ground truth" labels, assuming a single correct answer. However, human responses to creative content are diverse, forming a distribution of opinions rather than a consensus.

This research introduces a new benchmark for visual advertisement creativity, decomposing the concept into two measurable dimensions: atypicality and originality. The study investigates whether state-of-the-art Vision-Language Models (VLMs) can move beyond simple ranking to simulate the complex distribution of human perspectives and predict the level of disagreement a specific piece of content might provoke.

Methodology and Evaluation Tasks

The researchers shifted from binary classification toward Distribution Modeling. To simulate a collective human response, VLMs were prompted multiple times using high "temperature" settings to increase output randomness. This generated a spread of scores comparable to human rating distributions.

The models were evaluated on three primary tasks:

  1. Distribution Modeling: Measuring how closely the model's score distribution matches human ratings using KL Divergence.
  2. Disagreement Prediction: Assessing the model's ability to identify "polarizing" ads (those with high human rating variance).
  3. Pairwise Preference: Determining if the model can correctly identify the more creative ad when presented with two options.

The study benchmarked several open-source models, including LLaVA-7B, LLaVA-13B, and InternVL2-8B, alongside the closed-source GPT-4v.

Dataset Construction

To facilitate this evaluation, the authors developed two specialized datasets derived from the Pitt Advertisement Dataset:

  • Creative-100: This set contains 100 advertisements across 10 diverse categories (e.g., food, beauty, electronics). Each ad was annotated by 25 humans on a 1-3 scale for creativity, originality, and atypicality, providing a rich distribution of human sentiment.
  • Atypical-300: A larger collection of 300 ads specifically labeled for "atypicality"—the presence of unusual or physically impossible visual connections—designed to test a model's ability to detect visual anomalies.

Defining Creativity: Atypicality vs. Originality

The study adopts a marketing-centric framework where creativity is the sum of atypicality and originality.

  • Atypicality is considered an intrinsic property, such as a cow using a laptop, which relies on direct visual cues.
  • Originality is extrinsic, representing how much an ad deviates from others within its specific category.
Comparison of human and VLM rating distributions for sample ads, alongside a summary of performance metrics for creativity, originality, and atypicality.

Results and Discussion

The experimental results highlight a significant gap between ranking performance and distribution modeling.

Pairwise Ranking vs. Distribution Modeling

While models excel at ranking—with GPT-4v achieving an F1 score over 0.9 in pairwise comparisons—they struggle to replicate the nuanced "middle ground" of human opinion. Models often produce more extreme or uniform distributions compared to the varied responses of human annotators.

Model Performance and Reasoning

InternVL2-8B demonstrated the strongest alignment with average human ratings, reaching a Spearman’s correlation of 0.72 for creativity. Interestingly, smaller models sometimes outperformed larger ones in ranking tasks. This suggests that the reasoning capabilities of the underlying language model (e.g., Mistral) are more critical for evaluating creativity than raw parameter count.

The Disagreement Gap

A key finding is that current VLMs almost entirely fail to predict human disagreement. There was near-zero correlation between model-predicted variance and the actual standard deviation of human ratings. While models can recognize creative patterns, they lack the "human logic" required to understand why an image might be clever to one person but confusing to another.

Conclusion and Future Directions

The study concludes that while VLMs are powerful tools for identifying creative content, they do not yet capture the full spectrum of human subjectivity. The primary limitation remains dataset size, as collecting high-density human annotations is resource-intensive.

Future research should explore persona-based prompting—instructing models to respond as specific demographics (e.g., a "skeptical critic" or a "marketing executive")—to better simulate audience diversity. Additionally, providing models with broader cultural and brand context may improve their ability to assess extrinsic originality.

Summary updated

BibTeX
@misc{hou2025leveraging,
  author = {Zhaoyi Joey Hou and Adriana Kovashka and Xiang Lorraine Li},
  title = {{Leveraging Large Models for Evaluating Novel Content: A Case Study on Advertisement Creativity}},
  year = {2025},
  eprint = {2503.00046},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.00046},
}
Figure