Introduction
Traditional Image Quality Assessment (IQA) for AI-generated content often prioritizes technical fidelity—such as sharpness and the absence of artifacts—over communicative intent. However, in professional fields like advertising, the effectiveness of an image is determined by its ability to convey a specific message and evoke a targeted emotional response. To address this gap, researchers have introduced AIGI-VC, a novel dataset and framework designed to evaluate AI-generated images through the lens of visual communication.
The AIGI-VC Dataset
The AIGI-VC dataset serves as a benchmark for assessing "communicability" in AI-generated imagery. It comprises 2,500 images produced by five state-of-the-art generative models: Dall·E 3, Stable Diffusion XL, SD 2.0, SD 3.0, and Dreamlike Photoreal 2.0. The dataset is structured around 14 advertisement topics, ranging from commercial products like "Electronics" and "Baby Products" to public service themes such as "Animal Rights" and "Environment Protection."
Evaluation Dimensions: Clarity and Emotion
The study proposes two primary axes for quality assessment:
- Information Clarity: This metric evaluates whether the visual elements in the generated image accurately and clearly represent the components of the text prompt.
- Emotional Interaction: This metric assesses the image's ability to evoke specific human emotions, such as awe, amusement, or fear, as intended by the creative brief.
To establish a ground truth, the researchers employed a "Human-in-the-Loop" annotation process. This involved pairwise human preferences, initial reasoning generated by GPT-4o, and final rectification by human experts to create "Golden Descriptions" that explain the rationale behind quality judgments.
Benchmarking Results
The researchers evaluated 14 objective metrics and 7 Large Multi-modal Models (LMMs) on the AIGI-VC dataset. Key findings include:
- GPT-4o Superiority: GPT-4o emerged as the leading model, achieving approximately 79% accuracy in predicting human preferences for information clarity and providing the most coherent reasoning.
- Open-Source Performance Gap: Open-source models like LLaVA struggled significantly, often performing no better than random chance (~50% accuracy) in predicting communicative effectiveness.
- The Emotional Challenge: Across all models, performance on the "Emotional Interaction" dimension was consistently lower than on "Information Clarity," suggesting that AI still lacks a sophisticated understanding of human affect.
- Order Bias: Many LMMs exhibited "order bias," where their quality judgment changed depending on which image in a pair was presented first, highlighting a lack of robust evaluative consistency.
Conclusion
The AIGI-VC dataset highlights that aesthetic beauty does not equate to communicative success. While proprietary models like GPT-4o show promise in acting as automated design critics, there remains a significant gap in the ability of AI to grasp the emotional nuances of visual communication. Future work will look to expand this framework into other design domains and improve the reasoning capabilities of open-source models.
Summary updated