Abstract: Image composition recommendation aims to find the most aesthetically valuable crop in an image, assisting photographers in capturing balanced and professional-looking photos. Traditional neural network-based methods often rely on scoring pre-defined anchor boxes, which limits precision and computational efficiency. This paper proposes an end-to-end model that directly regresses optimal composition bounding boxes. By integrating a feature extraction module, a bounding box regression module, and a score regression module, the system overcomes the challenges of multi-modal aesthetic targets. Evaluated on the FCDB and FLMS datasets, the method achieves state-of-the-art performance in terms of Intersection over Union (IoU) and Boundary Displacement Error (Disp).
Introduction
Automating image composition—identifying the most aesthetically pleasing crop—is a complex task due to the inherent subjectivity of visual beauty. Most existing deep learning approaches utilize an anchor-based framework, generating thousands of candidate boxes and scoring them. This approach suffers from two primary limitations: resolution constraints (the final crop is restricted to the initial anchors) and high computational costs.
Direct bounding box regression offers a more flexible alternative but faces the "multi-modal" challenge: a single image often contains multiple valid compositions. Standard regression models tend to average these targets, resulting in mediocre outputs. This paper addresses these issues by partitioning the image into local regions and regressing refined coordinates within those regions.
Methodology
The proposed architecture moves beyond simple enumeration to a continuous geometric refinement process. The system is comprised of three core modules:
- Feature Extraction: A VGG16 backbone extracts high-level visual features from the input image.
- Local Crop Set Partitioning: To manage multiple valid compositions, the image is divided into an $8 \times 8$ grid, defining 225 local crop sets. This "divide and conquer" strategy ensures that within any specific local region, there is typically only one optimal crop, making the regression mathematically stable.
- Dual Regression Branches:
- Bounding Box Regression: For each of the 225 regions, the model uses RoI Align to extract local features and regresses precise coordinates ($d_{xi1}, d_{yi1}, d_{xi2}, d_{yi2}$).
- Score Regression: A parallel branch predicts an aesthetic score for each regressed box. The final recommendation is the box with the highest predicted score.
The model is trained using a multi-task loss function that combines Constraint Loss (keeping boxes within their local regions), $L_1$ Box Regression Loss, and a Score Distance Loss based on pairwise rankings.
Experimental Results
The model was trained on the CPC (Comparative Photo Composition) dataset and tested on two standard benchmarks: FCDB (321 images with single ground-truth crops) and FLMS (500 images with multiple human-annotated crops).
Quantitative Performance
The proposed method achieved state-of-the-art accuracy across both test sets:
- FCDB: Achieved an IoU of 0.6878, surpassing the previous best, GAIC (0.6802).
- FLMS: Reached an IoU of 0.8534, significantly outperforming VPN (0.8233).
- Efficiency: The system operates at 24 FPS, enabling real-time application in digital viewfinders.
Qualitative Comparison
Visual analysis demonstrates that the model produces crops that align more closely with professional aesthetic rules, such as the Rule of Thirds, compared to anchor-based methods that are limited by their initial proposals.
Conclusion
By shifting from anchor scoring to direct geometric regression within localized regions, this research provides a more precise and efficient solution for automated image composition. While the model excels in standard scenes, future work will focus on "subject-aware" features to better handle complex images with multiple distinct focal points and varying human aesthetic preferences.
Summary updated