AI-generated Image Quality Assessment in Visual Communication

Yu Tian, Yixuan Li, Baoliang Chen, Hanwei Zhu, Shiqi Wang, Sam Kwong

Introduction

Traditional Image Quality Assessment (IQA) for AI-generated content often prioritizes technical fidelity—such as sharpness and the absence of artifacts—over communicative intent. However, in professional fields like advertising, the effectiveness of an image is determined by its ability to convey a specific message and evoke a targeted emotional response. To address this gap, researchers have introduced AIGI-VC, a novel dataset and framework designed to evaluate AI-generated images through the lens of visual communication.

The AIGI-VC Dataset

The AIGI-VC dataset serves as a benchmark for assessing "communicability" in AI-generated imagery. It comprises 2,500 images produced by five state-of-the-art generative models: Dall·E 3, Stable Diffusion XL, SD 2.0, SD 3.0, and Dreamlike Photoreal 2.0. The dataset is structured around 14 advertisement topics, ranging from commercial products like "Electronics" and "Baby Products" to public service themes such as "Animal Rights" and "Environment Protection."

Overview of the AIGI-VC dataset, featuring 14 ad topics and 8 emotion categories across 2,500 generated images.

Evaluation Dimensions: Clarity and Emotion

The study proposes two primary axes for quality assessment:

  1. Information Clarity: This metric evaluates whether the visual elements in the generated image accurately and clearly represent the components of the text prompt.
  2. Emotional Interaction: This metric assesses the image's ability to evoke specific human emotions, such as awe, amusement, or fear, as intended by the creative brief.
The 14 advertisement topics included in the AIGI-VC dataset, spanning product-based and social welfare categories.

To establish a ground truth, the researchers employed a "Human-in-the-Loop" annotation process. This involved pairwise human preferences, initial reasoning generated by GPT-4o, and final rectification by human experts to create "Golden Descriptions" that explain the rationale behind quality judgments.

Benchmarking Results

The researchers evaluated 14 objective metrics and 7 Large Multi-modal Models (LMMs) on the AIGI-VC dataset. Key findings include:

  • GPT-4o Superiority: GPT-4o emerged as the leading model, achieving approximately 79% accuracy in predicting human preferences for information clarity and providing the most coherent reasoning.
  • Open-Source Performance Gap: Open-source models like LLaVA struggled significantly, often performing no better than random chance (~50% accuracy) in predicting communicative effectiveness.
  • The Emotional Challenge: Across all models, performance on the "Emotional Interaction" dimension was consistently lower than on "Information Clarity," suggesting that AI still lacks a sophisticated understanding of human affect.
  • Order Bias: Many LMMs exhibited "order bias," where their quality judgment changed depending on which image in a pair was presented first, highlighting a lack of robust evaluative consistency.

Conclusion

The AIGI-VC dataset highlights that aesthetic beauty does not equate to communicative success. While proprietary models like GPT-4o show promise in acting as automated design critics, there remains a significant gap in the ability of AI to grasp the emotional nuances of visual communication. Future work will look to expand this framework into other design domains and improve the reasoning capabilities of open-source models.

Summary updated

Abstract

Assessing the quality of artificial intelligence-generated images (AIGIs) plays a crucial role in their application in real-world scenarios. However, traditional image quality assessment (IQA) algorithms primarily focus on low-level visual perception, while existing IQA works on AIGIs overemphasize the generated content itself, neglecting its effectiveness in real-world applications. To bridge this gap, we propose AIGI-VC, a quality assessment database for AI-Generated Images in Visual Communication, which studies the communicability of AIGIs in the advertising field from the perspectives of information clarity and emotional interaction. The dataset consists of 2,500 images spanning 14 advertisement topics and 8 emotion types. It provides coarse-grained human preference annotations and fine-grained preference descriptions, benchmarking the abilities of IQA methods in preference prediction, interpretation, and reasoning. We conduct an empirical study of existing representative IQA methods and large multi-modal models on the AIGI-VC dataset, uncovering their strengths and weaknesses.
BibTeX
@inproceedings{tian2025ai,
  author = {Yu Tian and Yixuan Li and Baoliang Chen and Hanwei Zhu and Shiqi Wang and Sam Kwong},
  title = {{AI-generated Image Quality Assessment in Visual Communication}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i7.32795},
}
Figure