Q-REAL: Towards Realism and Plausibility Evaluation for AI-Generated Content

Shushi Wang, Zicheng Zhang, Chunyi Li, Wei Wang, Liya Ma, Fengjiao Chen, Xiaoyu Li, Xuezhi Cao, Guangtao Zhai, Xiaohong Liu

TL;DR

This paper introduces Q-Real, a new dataset and benchmark for fine-grained evaluation of AI-generated images, focusing on "naturalness" (realism) and "distortion" (plausibility). They propose a novel three-stage training pipeline for Multimodal Large Language Models (MLLMs) to perform this fine-grained assessment, significantly improving their ability to identify and explain quality issues.

What Problem Does This Paper Address?

The rapid growth of AI-generated content, particularly images, necessitates robust evaluation methods. Current approaches often provide a single, coarse quality score, which is insufficient for pinpointing specific issues and guiding model improvements. This "black box" evaluation makes it hard to understand why an image is considered low quality. The paper argues that for photorealistic AI image generation, two key aspects are crucial: naturalness (how much it looks like a real-world photo, avoiding the "AI look") and distortion (physical implausibilities, blurriness, or structural errors). Existing methods struggle to assess these dimensions with the granularity needed for effective model optimization.

Methods

The authors introduce a multi-stage approach:

  1. Q-Real Dataset Construction: They collected 10,000 AI-generated images, combining existing datasets with new generations from recent models. Crucially, they annotate major entities within each image using bounding boxes. For each entity, they provide objective judgment questions and detailed descriptive analyses for both naturalness and distortion. This annotation process involves a hybrid automated and manual approach for efficiency and quality.
  2. Q-Real Bench: This benchmark is built on the Q-Real dataset and comprises two tasks:
  3. ObjectQA: Models must answer binary (Yes/No) judgment questions about the naturalness and distortion of individual entities.
  4. ImageQA: Models must localize problematic entities with bounding boxes and provide detailed descriptions of the issues.
  5. Three-Stage Progressive Training Pipeline for MLLMs:
    • Stage 1 (Object-Level Judgment Training): Fine-tuning MLLMs on question-answer pairs related to entity-level naturalness and distortion.
  6. Stage 2 (Fine-Grained Evaluation Capability Training): Training MLLMs to localize problematic entities and generate detailed reasoning descriptions.
  7. Stage 3 (Quality Score Prediction Training): Using a Chain-of-Thought (CoT) prompt that incorporates the fine-grained analysis from Stage 2 to predict an overall quality score. This stage also involves a novel scoring method that maps numerical MOS values to discrete adjective ratings.

Datasets

  • Q-Real Dataset: A newly introduced dataset of 10,000 AI-generated images with fine-grained annotations for naturalness and distortion, including entity localization and descriptive reasoning.
  • Q-Eval-100K: A subset of this existing dataset was used as a source for Q-Real.
  • AGIQA-3K and ImageReward: Used for evaluating the proposed quality assessment model.
  • RichHF-18K: Used for cross-dataset validation.

Key Insights and Ideas

  • Fine-grained evaluation is essential: Moving beyond single scores is critical for understanding and improving AI image generation.
  • Naturalness and Distortion are key dimensions: These two aspects capture the most important quality aspects for photorealistic AI images.
  • Entity-level annotation is powerful: Localizing and analyzing specific entities provides much richer information than image-level scores.
  • MLLMs are well-suited for this task: With appropriate training, MLLMs can effectively perform complex visual reasoning and description generation.
  • A staged training approach is effective: Progressively building from object-level judgment to detailed analysis and scoring leads to superior performance.
  • Chain-of-Thought prompting enhances evaluation: Incorporating detailed reasoning into prompts significantly improves the accuracy of quality assessment.

Key Results

  • Q-Real Bench Performance: Fine-tuned MLLMs (specifically Qwen3-VL-8B) significantly outperform zero-shot baselines and even closed-source models like GPT-40 on both ObjectQA and ImageQA tasks.
  • On ObjectQA, fine-tuned models showed improvements of up to 57.3% in accuracy and 44.5% in F1 score.
  • On ImageQA, fine-tuned models achieved detection rates above 0.5 and description similarity (LLM Score) exceeding 0.7 for naturalness, and 0.5 and 0.6 respectively for distortion.
  • Quality Assessment: The proposed Q-Real-Score model achieved state-of-the-art performance on Q-Real Bench, AGIQA-3K, and ImageReward datasets, outperforming competitors like Q-Eval-Score.
  • Ablation Studies:
    • Stage 1 training was crucial for preparing models for the more complex Stage 2 training.
  • Including both naturalness and distortion issues in the CoT prompt for Stage 3 significantly improved quality prediction.
  • Training-inference consistency in the prompt design had the most significant positive impact.

Limitations and Future Work

  • Limitations: The paper doesn't explicitly detail limitations beyond the inherent challenges of subjective quality assessment and the reliance on MLLMs. The performance on distortion is noted as generally weaker than naturalness, indicating room for improvement.
  • Future Work: The authors hope their work inspires future advancements in MLLM-based quality assessment for AI-generated content. The dataset and code are to be released, which will facilitate further research in this area. The staged training pipeline itself is a promising direction for developing more capable evaluation models.

Summary updated

Abstract

Quality assessment of AI-generated content is crucial for evaluating model capability and guiding model optimization. However, most existing quality assessment datasets and models provide only a single quality score, which is too coarse to offer targeted guidance for improving generative models. In current applications of AI-generated images, realism and plausibility are two critical dimensions, and with the emergence of unified generation-understanding models, fine-grained evaluation along these dimensions becomes especially effective for improving generative performance. Therefore, we introduce Q-Real, a novel dataset for fine-grained evaluation of realism and plausibility in AI-generated images. Q-Real consists of 3,088 images generated by popular text-to-image models. For each image, we annotate the locations of major entities and provide a set of judgment questions and attribution descriptions for these along the dimensions of realism and plausibility. Considering that recent advances in multi-modal large language models (MLLMs) enable fine-grained evaluation of AI-generated images, we construct Q-Real Bench to evaluate them on two tasks: judgment and grounding with reasoning. Finally, to enhance MLLM capabilities, we design a fine-tuning framework and conduct experiments on multiple MLLMs using our dataset. Experimental results demonstrate the high quality and significance of our dataset and the comprehensiveness of the benchmark. Dataset and code will be released upon publication.
BibTeX
@misc{wang2025q,
  author = {Shushi Wang and Zicheng Zhang and Chunyi Li and Wei Wang and Liya Ma and Fengjiao Chen and Xiaoyu Li and Xuezhi Cao and Guangtao Zhai and Xiaohong Liu},
  title = {{Q-REAL: Towards Realism and Plausibility Evaluation for AI-Generated Content}},
  year = {2025},
  eprint = {2511.16908},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.16908},
}
Figure