When a human evaluates a photograph or a painting, they rarely stop at a single summary judgment like "7 out of 10." Instead, they provide a critique: "The lighting is dramatic, but the composition feels cluttered," or "The subject is sharp, but the background blur is unnatural."
For years, automated Image Quality Assessment (IQA) and Image Aesthetic Assessment (IAA) models operated in direct opposition to this human tendency. Trained on mean opinion scores, these deep learning models functioned as black boxes, compressing the rich, multi-dimensional nature of human visual perception into a single scalar value.
While a single numerical score is useful for sorting or ranking, it suffers from an interpretability gap. As noted in a comprehensive survey by Min et al., scalar scores fail to diagnose why an image or video degrades, making it difficult for engineers to troubleshoot compression codecs or transmission pipelines Perceptual video quality assessment: a survey. Similarly, Lo et al. point out that a raw score provides no actionable guidance to a photographer looking to improve their shot Intelligent Photographing Interface with On-Device Aesthetic Quality Assessment.
To bridge this gap, a major research direction has emerged to make visual quality assessment interpretable. By decomposing global judgments into explicit attributes, mapping quality across spatial dimensions, and leveraging multimodal language models, modern frameworks explain the underlying logic of aesthetic and quality decisions.
Decomposing Aesthetics into Explicit Attributes
The first intuitive step away from black-box scoring is to break down an image's aesthetic value into explicit, human-understandable sub-components or photographic rules.
Instead of mapping an image directly to a single score, models can be designed to evaluate distinct dimensions simultaneously. Jin et al. formalized this by introducing multi-attribute networks that score specific categories such as lighting, composition, focus, subject matter, and camera use Aesthetic Attributes Assessment of Images.
Opaque Model: [ Image ] ───────────────────────────► [ 6.5 / 10 ]
Attribute Model: [ Image ] ───► [ Lighting: 8.2 ] ──► [ Final Score: 6.5 ]
[ Composition: 4.1 ]
[ Focus: 7.5 ]
A core challenge of this approach is data scarcity: annotating thousands of images for multiple distinct aesthetic attributes requires significant expert labor. To bypass this bottleneck, Huang et al. demonstrated that a model can learn to predict detailed attribute scores using only global aesthetic labels Predicting Scores of Various Aesthetic Attribute Sets by Learning from Overall Score Labels. By leveraging pre-trained feature extractors from auxiliary tasks—such as object detection for subject emphasis or depth estimation for focus—the framework disentangles the global score into its constituent parts without explicit attribute training data.
Beyond statistical attributes, models can explicitly incorporate established photographic grammar. For example, Hong et al. developed CACNet, which explicitly models nine classic composition rules, such as the rule-of-thirds and diagonal lines Composing Photos Like a Photographer. By forcing the network to justify its cropping decisions through these geometric primitives, the system moves from implicit data-fitting to rule-based explanation.
Spatial Grounding: Mapping Quality in Two Dimensions
Visual degradations and aesthetic triumphs are rarely uniform; they happen in specific regions of an image. A single scalar score completely loses this spatial context. A second core mechanism for interpretability is therefore spatial grounding—forcing models to generate dense, two-dimensional maps that pinpoint exactly where an image succeeds or fails.
In image quality assessment, Pan et al. introduced a no-reference model that predicts a pixel-by-pixel quality map Blind Predicting Similar Quality Map for Image Quality Assessment. By using a generative network to "hallucinate" a local distortion map before pooling it into a final score, the model reveals whether a low rating is caused by a localized patch of sensor noise or a global compression blur.
[ Input Image ] ──► [ Generative U-Net ] ──► [ Spatial Quality Map ] ──► [ Global Score ]
(Visualizes exactly
where blur/noise occurs)
This spatial awareness is equally vital for automated image cropping and re-composition. Niu et al. developed ASM-Net, which outputs an aesthetic score map based on spatial composition and visual saliency Image Cropping with Composition and Saliency Aware Aesthetic Score Map. Similarly, the authors of Re-Compose the Image by Evaluating the Crop on More Than Just a Score utilized multi-layer score maps. These maps explicitly separate the attractive elements inside a proposed crop from the distracting elements left outside it, providing a clear visual rationale for why one crop boundary is superior to another.
Linguistic Critiques via Vision-Language Models
The most recent and expressive paradigm shift in this field leverages Multimodal Large Language Models (MLLMs) to translate visual quality directly into natural language descriptions. Language serves as the ultimate interpretable proxy, capable of blending spatial, attribute, and holistic reasoning into a single output.
Instead of outputting a number, models like DepictQA and DepictQA-Wild, developed by You et al., are trained to identify, rate, and textually reason about image artifacts Depicting Beyond Scores: Advancing Image Quality Assessment Through Multi-modal Language Models, Descriptive Image Quality Assessment in the Wild. These models can explicitly articulate how a specific distortion interacts with the semantic content of an image—for instance, explaining that high-frequency noise has obliterated the texture of a distant subject.
To train these linguistic evaluators, researchers have shifted from collecting numerical ratings to curating rich text datasets:
- Real-World Critiques: Nieto et al. introduced the Reddit Photo Critique Dataset (RPCD), capturing thousands of high-resolution images paired with natural language feedback from online photography communities Understanding Aesthetics with Language: A Photo Critique Dataset for Aesthetic Assessment.
- Expert Fine-Art Annotations: For complex artistic domains like paintings and drawings, Jin et al. introduced APDDv2, which pairs expert-labeled scores across ten distinct aesthetic attributes with detailed textual comments APDDv2: Aesthetics of Paintings and Drawings Dataset with Artist Labeled Scores and Comments.
To make these language-driven assessments reliable, researchers have moved toward modular, agentic architectures. As open-source language models can struggle with fine-grained visual discrimination, Zhu et al. proposed AgenticIQA AgenticIQA: An Agentic Framework for Adaptive and Interpretable Image Quality Assessment. This framework uses a Vision-Language Model as a central planner that dynamically invokes specialized, low-level IQA tools. A summarizer then synthesizes the tools' quantitative outputs into a cohesive, human-aligned textual explanation.
Why It Works: The Intuition of Shared Semantics
Why does forcing a model to generate text, maps, or attributes improve our systems? The answer lies in multi-task regularization and semantic grounding.
When a deep neural network is trained solely to predict a single scalar score, it often optimizes for superficial statistical shortcuts—such as global contrast, color histograms, or high-frequency edge density. The model might achieve high accuracy on a specific test set, but it does not "understand" aesthetics or quality; it has merely memorized a correlation.
By forcing the model to co-estimate spatial maps, discrete photographic attributes, or natural language descriptions, the learning problem changes fundamentally. The network must map the input image into a structured, human-aligned semantic space before it can output a judgment. This shared representation acts as a powerful regularizer: a model cannot easily spoof a high aesthetic score if it is simultaneously required to prove, via text or a spatial map, that the composition adheres to the rule-of-thirds and that the lighting is well-balanced. Grounding the evaluation in explicit human concepts forces the model's internal representations to align with actual human perceptual priorities.
Key Foundational Work
Intelligent Photographing Interface with On-Device Aesthetic Quality Assessment (2012) — contributed 1 concept elements to this direction.
Learning to Play Like the Great Pianists (2005) — contributed 1 concept elements to this direction.