Predicting Scores of Various Aesthetic Attribute Sets by Learning from Overall Score Labels

Heng Huang, Xin Jin, Yaqi Liu, Hao Lou, Chaoen Xiao, Shuai Cui, Xining Li, Dongqing Zou

TL;DR

The F2S (Attribute Features to Attribute Scores) model predicts detailed aesthetic attribute scores (like lighting, composition, and object emphasis) for images using only the "overall" aesthetic score as a training label. By leveraging pre-trained feature extractors from diverse computer vision tasks (e.g., object detection and depth estimation), the framework bypasses the need for expensive, expert-level attribute annotations while providing interpretable feedback on why a photo is considered "good."

What Problem Does This Paper Address?

In Image Aesthetic Assessment (IAA), we don't just want to know if a photo is beautiful; we want to know why. However, training models to provide this level of detail is difficult because high-quality datasets with attribute-level labels (e.g., "Rule of Thirds: 8/10") are rare. Annotating these attributes requires professional photographers, making large-scale data collection nearly impossible.

Most existing datasets only provide a single "overall" score. The authors address this gap by asking: Can we learn to predict individual aesthetic attributes using only the overall score as a guide?

Methods

The authors introduce the F2S framework, which treats aesthetic attributes as a combination of specialized features and a learnable "contribution" weight.

  1. Attribute Feature Extraction: Instead of learning features from scratch, the model "borrows" pre-trained networks from other domains to act as proxies for aesthetic attributes:
  2. Content: Uses MaskFormer (scene segmentation).
  3. Object Emphasis: Uses DETR (object detection).
  4. Light: Uses a technique from illumination estimation.
  5. Rule of Thirds (RoT): Uses a context-aware line detector (DHT).
  6. Depth of Field (DoF): Uses Monodepth2 (self-supervised depth prediction).
  7. The Contribution Module: Not every attribute matters equally for every photo. The model uses an attention-based module to calculate Attribute Contributions ($C_i$). For example, in a sunset photo, "vivid color" might contribute more to the overall score than "human pose."
  8. Weakly Supervised Learning: The model is trained using a composite loss function. It predicts individual attribute scores ($S_i$) and combines them with their contribution weights ($C_i$) to estimate the overall score. The model is optimized to make this estimated overall score match the ground truth, effectively "back-propagating" the overall score's signal into the individual attribute branches.

Datasets

The researchers evaluated their model across several major aesthetic benchmarks:

  • AVA: The largest aesthetic dataset (255k+ images), used here for fine-tuning and portrait extraction.
  • AADB: Contains 10,000 images with 11 attribute labels; used to compare the F2S model against supervised methods.
  • EVA: A dataset with 4,070 images focusing on visual, composition, quality, and semantic scores.
  • PCCD: Used for its specific "focus" and "camera use" attributes.
  • AVA-Portraits: A custom subset of 4,500 images curated by the authors to test specialized attributes like "face lighting" and "pose."

Key Insights and Ideas

  • Cross-Task Synergy: The "aha moment" is that features learned for objective tasks (like "where is the cat?" or "how deep is this scene?") are highly correlated with subjective aesthetic judgments. A model that knows how to segment a scene already has the "vocabulary" needed to judge its composition.
  • The "Extra" Attribute: To account for aesthetic factors not covered by their specific modules, the authors include an "Extra Score" branch. This ensures the model doesn't try to force-fit all aesthetic quality into a limited set of pre-defined attributes.
  • Aesthetics are Weighted: By defining the overall score as a weighted sum of attributes, the model acknowledges that aesthetics are contextual. This allows the model to explain why it gave a certain score (e.g., "This photo has a high score primarily because of its Vivid Color and Rule of Thirds").

Key Results

  • Comparable to Supervised Models: On the AADB dataset, the semi-supervised F2S model achieved an overall score SRCC (Spearman’s Rank Correlation Coefficient) of 0.728, nearly identical to the 0.730 achieved by a fully supervised model trained with attribute labels.
  • High Correlation: Even without seeing attribute labels during training, the F2S model's predicted scores for "Content," "Color Harmony," and "Light" showed high correlation with human-annotated ground truths.
  • Visual Validation: Using Grad-CAM (a visualization tool), the authors showed that their semi-supervised model "looks" at the same parts of an image as a supervised model. For "Object Emphasis," both models correctly focused on the central subject of the photo.

Limitations and Future Work

  • Expertise in Selection: While the model doesn't need attribute labels, it still requires a human expert to choose the "right" pre-trained feature extractor for a given attribute.
  • Feature Dependency: The accuracy of the aesthetic prediction is capped by the quality of the underlying task-specific models (e.g., if the depth estimator fails, the Depth of Field score will be inaccurate).
  • Future Directions: The authors suggest expanding the framework to specialized domains like sports photography or landscapes, where unique aesthetic rules (like "peak action" or "horizon placement") apply.

Summary updated

Abstract

Now many mobile phones embed deep-learning models for evaluation or guidance on photography. These models cannot provide detailed results like human pose scores or scene color scores because of the rare of corresponding aesthetic attribute data. However, the annotation of image aesthetic attribute scores requires experienced artists and professional photographers, which hinders the collection of large-scale fully-annotated datasets. In this paper, we propose to replace image attribute labels with feature extractors. First, a novel aesthetic attribute evaluation framework based on attribute features is proposed to predict attribute scores and overall scores. We call it the F2S (attribute features to attribute scores) model. We use networks from different tasks to provide attribute features to our F2S models. Then, we define an aesthetic attribute contribution to describe the role of aesthetic attributes throughout an image and use it with the attribute scores and the overall scores to train our F2S model. Sufficient experiments on publicly available datasets demonstrate that our F2S model achieves comparable performance with those trained on the datasets with fully-annotated aesthetic attribute score labels. Our method makes it feasible to learn meaningful attribute scores for various aesthetic attribute sets in different types of images with only overall aesthetic scores.
BibTeX
@inproceedings{huang2024predicting,
  author = {Heng Huang and Xin Jin and Yaqi Liu and Hao Lou and Chaoen Xiao and Shuai Cui and Xining Li and Dongqing Zou},
  title = {{Predicting Scores of Various Aesthetic Attribute Sets by Learning from Overall Score Labels}},
  booktitle = {MCGE@MM},
  year = {2024},
  doi = {10.1145/3688867.3690174},
}

Loading related content...

Figure