Traditional Image Quality Assessment (IQA) and Video Quality Assessment (VQA) metrics were built for a world dominated by camera noise, motion blur, and compression artifacts. In that paradigm, "degradation" meant a loss of signal fidelity—a blurring of edges or an introduction of pixel-level grain.
However, the rise of deep generative models (such as GANs and diffusion models) has broken these underlying assumptions. Generative models do not merely degrade signals; they hallucinate new ones. This shift introduces unique structural and semantic artifacts that traditional metrics are fundamentally unequipped to detect.
The Failure of Traditional Metrics
To understand why traditional metrics fail, we must look at how generative models optimize for visual appeal. In image restoration and generation, there is a well-documented perception-distortion tradeoff. As models attempt to produce highly realistic, sharp textures, they often deviate significantly from the exact pixel layout of a reference image.
Standard full-reference metrics like PSNR and SSIM penalize these spatial shifts and granular variations, misinterpreting creative or slightly shifted textures as poor quality. Conversely, as Manri Cheon et al. and Shuwei Shi et al. observe, modern generative networks can create high-frequency details that appear perfectly sharp to simple mathematical metrics but are entirely unrealistic or inconsistent upon human inspection.
Traditional Paradigm:
[Original Image] ---> (Noise / Compression) ---> [Degraded Image]
*Evaluated by measuring pixel deviation or high-frequency loss.*
Generative Paradigm:
[Text Prompt / Reference] ---> (Generative Model) ---> [Synthesized Image]
*Can be perfectly sharp and clean at the pixel level, yet contain structural hallucinations.*
For no-reference (blind) assessment, the problem is equally acute. Traditional blind metrics look for natural scene statistics or mathematical blur. As Shuyang Gu et al. point out, these tools cannot identify the "checkerboard" patterns, unnatural structural layouts, or unreasonable visual patterns specific to neural network outputs. An image generated by a diffusion model might have zero camera noise and perfect contrast, yet depict a person with six fingers or a house with impossible geometry. To a traditional metric, this image is flawless; to a human, it falls straight into the uncanny valley.
The Anatomy of Generative Artifacts
To evaluate AI-generated content (AIGC) accurately, researchers have shifted toward isolating and benchmarking the specific failure modes of generative architectures. These artifacts span multiple dimensions:
- Structural and Anatomical Inconsistencies: Generative models frequently struggle with global spatial layouts and fine-grained geometry. Zhiyu Jin et al. note that sub-optimal generation parameters can cause structural failures like repetitive object artifacts or disordered spatial layouts. Furthermore, as shown by Zicheng Zhang et al., even advanced automated models often overlook blatant anatomical errors (such as incorrect digit counts) that human observers spot instantly.
- Text-to-Image Mismatch: A high-quality synthetic image must not only look realistic but also faithfully execute its conditioning prompt. Chunyi Li et al. demonstrate that sub-optimal parameters (like improper Classifier-Free Guidance scales) introduce unique distortions and prompt-alignment failures that require specialized evaluation frameworks.
- Temporal and Physical Violations: In video generation, artifacts extend into the time domain. Bowen Qu et al. and Zijian Chen et al. highlight that AI-generated videos frequently suffer from content drift, objects morphing unnaturally, and blatant violations of physical laws (such as objects defying gravity). Traditional VQA models optimized for transmission or compression noise completely miss these semantic and logical failures.
To capture these nuances, the community has developed large-scale datasets containing authentic generative distortions paired with human subjective scores, such as PIPAL (Jinjin Gu et al.), AGIQA-1K (Zicheng Zhang et al.), and VDPVE (Xiaohong Liu et al.).
Shifting to Semantic and Multimodal Evaluation
Because generative artifacts are semantic and structural rather than purely signal-based, modern quality assessment frameworks leverage the power of vision-language models (VLMs) and Large Multimodal Models (LMMs). The core intuition is that an evaluator needs "common sense" and cross-modal understanding to judge whether an image or video makes sense.
One prominent methodology relies on vision-language correspondence using models like CLIP. By embedding both the generated image and text prompts into a shared semantic space, models can evaluate quality as a multi-dimensional property. For example, the AMFF-Net framework proposed by Tianwei Zhou et al. explicitly decouples the evaluation into three dimensions: technical visual quality, authenticity (realism), and text-to-image consistency.
[AI-Generated Image] ───► [Multi-Scale Feature Extractor] ───┐
├───► [Perceptual Quality Score]
[Text Prompt] ───► [CLIP Semantic Embedding] ───┘
Similarly, Jun Fu et al. utilize learnable multi-modal prompts within CLIP, treating the alignment between text and visual features as a primary proxy for overall perceptual quality. For completely blind scenarios where no text prompt or reference image is available, researchers like Yixuan Li et al. have designed models that fuse shape-biased and texture-biased deep features, mimicking the human visual system's dual reliance on structural integrity and stylistic consistency to detect unnaturalness.
The latest frontier involves fine-tuning LMMs to act as automated human proxies. By pairing traditional deep learning backbones with the logical reasoning capabilities of language models, frameworks like MA-AGIQA (Puyi Wang et al.) can evaluate both low-level visual fidelity and high-level conceptual logic. To ensure these large models align with actual human perception, projects like EVALALIGN (Zhiyu Tan et al.) use supervised fine-tuning on human-aligned feedback datasets, bridging the domain gap that occurs when evaluation models trained purely on natural photography try to judge the logical boundaries of synthetic media.
Key Foundational Work
GIQA: Generated Image Quality Assessment (2020) — contributed 2 concept elements to this direction.
PIPAL: A Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration (2020) — contributed 1 concept elements to this direction.
Perceptual Image Quality Assessment With Transformers (2021) — contributed 1 concept elements to this direction.