TL;DR
The F2S (Attribute Features to Attribute Scores) model predicts detailed aesthetic attribute scores (like lighting, composition, and object emphasis) for images using only the "overall" aesthetic score as a training label. By leveraging pre-trained feature extractors from diverse computer vision tasks (e.g., object detection and depth estimation), the framework bypasses the need for expensive, expert-level attribute annotations while providing interpretable feedback on why a photo is considered "good."
What Problem Does This Paper Address?
In Image Aesthetic Assessment (IAA), we don't just want to know if a photo is beautiful; we want to know why. However, training models to provide this level of detail is difficult because high-quality datasets with attribute-level labels (e.g., "Rule of Thirds: 8/10") are rare. Annotating these attributes requires professional photographers, making large-scale data collection nearly impossible.
Most existing datasets only provide a single "overall" score. The authors address this gap by asking: Can we learn to predict individual aesthetic attributes using only the overall score as a guide?
Methods
The authors introduce the F2S framework, which treats aesthetic attributes as a combination of specialized features and a learnable "contribution" weight.
- Attribute Feature Extraction: Instead of learning features from scratch, the model "borrows" pre-trained networks from other domains to act as proxies for aesthetic attributes:
- Content: Uses MaskFormer (scene segmentation).
- Object Emphasis: Uses DETR (object detection).
- Light: Uses a technique from illumination estimation.
- Rule of Thirds (RoT): Uses a context-aware line detector (DHT).
- Depth of Field (DoF): Uses Monodepth2 (self-supervised depth prediction).
- The Contribution Module: Not every attribute matters equally for every photo. The model uses an attention-based module to calculate Attribute Contributions ($C_i$). For example, in a sunset photo, "vivid color" might contribute more to the overall score than "human pose."
- Weakly Supervised Learning: The model is trained using a composite loss function. It predicts individual attribute scores ($S_i$) and combines them with their contribution weights ($C_i$) to estimate the overall score. The model is optimized to make this estimated overall score match the ground truth, effectively "back-propagating" the overall score's signal into the individual attribute branches.
Datasets
The researchers evaluated their model across several major aesthetic benchmarks:
- AVA: The largest aesthetic dataset (255k+ images), used here for fine-tuning and portrait extraction.
- AADB: Contains 10,000 images with 11 attribute labels; used to compare the F2S model against supervised methods.
- EVA: A dataset with 4,070 images focusing on visual, composition, quality, and semantic scores.
- PCCD: Used for its specific "focus" and "camera use" attributes.
- AVA-Portraits: A custom subset of 4,500 images curated by the authors to test specialized attributes like "face lighting" and "pose."
Key Insights and Ideas
- Cross-Task Synergy: The "aha moment" is that features learned for objective tasks (like "where is the cat?" or "how deep is this scene?") are highly correlated with subjective aesthetic judgments. A model that knows how to segment a scene already has the "vocabulary" needed to judge its composition.
- The "Extra" Attribute: To account for aesthetic factors not covered by their specific modules, the authors include an "Extra Score" branch. This ensures the model doesn't try to force-fit all aesthetic quality into a limited set of pre-defined attributes.
- Aesthetics are Weighted: By defining the overall score as a weighted sum of attributes, the model acknowledges that aesthetics are contextual. This allows the model to explain why it gave a certain score (e.g., "This photo has a high score primarily because of its Vivid Color and Rule of Thirds").
Key Results
- Comparable to Supervised Models: On the AADB dataset, the semi-supervised F2S model achieved an overall score SRCC (Spearman’s Rank Correlation Coefficient) of 0.728, nearly identical to the 0.730 achieved by a fully supervised model trained with attribute labels.
- High Correlation: Even without seeing attribute labels during training, the F2S model's predicted scores for "Content," "Color Harmony," and "Light" showed high correlation with human-annotated ground truths.
- Visual Validation: Using Grad-CAM (a visualization tool), the authors showed that their semi-supervised model "looks" at the same parts of an image as a supervised model. For "Object Emphasis," both models correctly focused on the central subject of the photo.
Limitations and Future Work
- Expertise in Selection: While the model doesn't need attribute labels, it still requires a human expert to choose the "right" pre-trained feature extractor for a given attribute.
- Feature Dependency: The accuracy of the aesthetic prediction is capped by the quality of the underlying task-specific models (e.g., if the depth estimator fails, the Depth of Field score will be inaccurate).
- Future Directions: The authors suggest expanding the framework to specialized domains like sports photography or landscapes, where unique aesthetic rules (like "peak action" or "horizon placement") apply.
Summary updated