Abstract
Traditional Image Quality Assessment (IQA) focuses on low-level technicalities like noise and aesthetics, often neglecting the functional effectiveness of images in real-world applications. This paper introduces AIGI-VC, the first dataset specifically designed to evaluate AI-generated images (AIGIs) in the context of visual communication, such as advertising. The dataset comprises 2,500 images generated by five state-of-the-art models, annotated for information clarity and emotional resonance. Our benchmarking reveals that while models like GPT-4o lead in performance, a significant gap remains between generating "pretty" images and creating effective communicative tools.
Introduction
In the realm of advertising and marketing, the quality of an image is defined by its ability to convey specific information and evoke intended emotions. However, current IQA metrics and Large Multi-modal Models (LMMs) are not typically tuned to these nuances. A "high-quality" advertisement must achieve two primary goals: Information Clarity (clearly depicting all prompted elements) and Emotional Interaction (powerfully evoking emotions like "awe" or "excitement"). To address the lack of benchmarks for these functional aspects, we present the AIGI-VC dataset.
The AIGI-VC Dataset
The AIGI-VC dataset was constructed to bridge the gap between generative capability and practical application. It consists of 2,500 images generated from 500 distinct prompts using five leading models: DALL·E 3, Stable Diffusion XL, SD 3.0, SD 2.0, and Dreamlike Photoreal 2.0.
The prompts cover 14 diverse advertising topics and 8 specific emotion categories based on the Mikels model. Unlike general-purpose datasets, AIGI-VC includes specialized annotations for Rationality and Emotional Interaction, providing a granular look at how AI handles complex visual communication tasks.
Methodology
The researchers employed a structured pipeline for dataset creation and annotation:
- Prompt Engineering: GPT-4V designed prompts across topics like Animal Rights, Cosmetics, and Electronics, each paired with a target emotion.
- Human-in-the-Loop Annotation: To ensure high-quality labels, a two-tiered approach was used. For coarse-grained rankings, Maximum A Posteriori (MAP) estimation was applied to pairwise comparisons, reducing manual labor by 60%. For fine-grained reasoning, human experts verified "pseudo-reference" descriptions generated by GPT-4o to create "golden descriptions."
- Benchmarking: The study evaluated 14 objective metrics and several LMMs, including LLaVA, Qwen-VL, and GPT-4o, against human preferences.
Key Insights and Results
The benchmarking process yielded several critical findings regarding the current state of AI in visual communication:
- The Communication Gap: High aesthetic scores do not guarantee communicative effectiveness. Models often produce visually pleasing images that fail to evoke the specific intended emotion.
- Emotional Interaction is a Bottleneck: Models generally perform better at "Information Clarity" than "Emotional Interaction." Accurately judging and evoking human emotion remains a significant hurdle for current AI.
- LMM Performance and Bias: GPT-4o emerged as the top performer, achieving ~79% accuracy in preference prediction. However, open-source LMMs like LLaVA showed significant inconsistency, with their preferences often shifting based on the order of image presentation.
- Metric Efficacy: CLIP-based metrics like PickScore and HPSv2 outperformed traditional IQA metrics (e.g., HyperIQA), which struggle with the unique artifacts and characteristics of AI-generated content.
Conclusion and Future Directions
While GPT-4o serves as a strong baseline for evaluating visual communication, it still falls short of human expert precision. The AIGI-VC dataset highlights the need for future IQA metrics to evolve beyond pixel-level quality toward multi-dimensional assessments that incorporate human psychological and communicative responses. Future work should also address the inherent subjectivity of emotion and potential biases in using LMMs as both evaluators and data generators.
Summary updated