Re-Compose the Image by Evaluating the Crop on More Than Just a Score

Yang Cheng, Qian Lin, Jan P. AllebachWACV 2022

TL;DR

This paper introduces a novel "10-layer aesthetic score map" that goes beyond a single score to explain why a particular image crop is good, by considering the position of salient objects relative to the crop region. This enriched ground truth allows for a more interpretable and effective automatic image re-composition system, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Taking photos with mobile devices is ubiquitous, but often results in images with unwanted elements or suboptimal aspect ratios. Re-composing these images to improve framing and composition is a crucial post-processing step. However, this task is highly subjective, relying on aesthetic taste, making it challenging for inexperienced users and time-consuming for large photo collections. Existing automatic re-composition methods often provide a single score for a crop but lack explainability, failing to articulate why a crop is considered good or bad. Furthermore, they often overlook how the position of salient objects (both points of interest and distractions) within the crop region influences the overall aesthetic quality.

Methods

The authors propose an automatic image re-composition system with two main components: crop candidate proposal and crop candidate evaluation.

  • Crop Candidates Proposal: This uses a standard sliding window algorithm to generate various crop candidates of different scales and aspect ratios for a given desired output aspect ratio.
  • Crop Candidate Evaluation: This is the core contribution. Instead of relying on a single score, they introduce a 10-layer aesthetic score map. This map encodes how the saliency of objects in the original image, relative to the crop region, contributes to the crop's quality.
  • The 10 regions are defined by dividing the crop into 9 subregions (based on the rule of thirds) and a 10th region representing the area outside the crop.
  • A ground-truth 10-layer score map is generated by analyzing existing datasets, where each crop's score and saliency map are used to assign values to the corresponding regions. For instance, a distracting element outside the crop would contribute to the 10th layer, while a point of interest inside the crop would contribute to the inner layers.
  • A neural network, built upon MobileNetV2 with an hourglass network, is trained to predict both this 10-layer score map and a final crop score. RoI Align and RoD Align operators are used to effectively extract features from both the image and the predicted score map.

Datasets

  • GAIC Dataset [19]: Used for training and evaluation. It contains approximately 90 crops per image, rated by 19 recruited subjects (Mean Opinion Score - MOS). The authors use 1037 images for training and the rest for testing.
  • Own Dataset: A custom dataset of 100 images (50 with main objects, 50 landscapes) was collected and evaluated by 17 subjects to assess subjective preference.

Key Insights and Ideas

  • Explainability through Saliency Location: The most significant insight is that the position of salient objects (both attractive and distracting) relative to the crop boundary is as crucial as their presence. The 10-layer score map provides a nuanced understanding of this relationship.
  • Distinguishing Attractions from Distractions: The 10th layer of the score map is specifically designed to capture elements that are outside the crop. By analyzing saliency in this layer, the model can effectively identify and penalize crops that include distractions.
  • Enriching Ground Truth: The paper highlights the limitations of existing datasets that only provide a single score. By generating a richer ground truth (the 10-layer score map), they enable a more robust and interpretable learning process for automatic cropping.
  • Rule of Thirds Integration: The method implicitly incorporates the rule of thirds by dividing the crop into 9 regions, aligning with established photographic composition principles.

Key Results

  • State-of-the-Art Performance: The proposed method achieves state-of-the-art performance on the GAIC dataset, outperforming several existing methods across metrics like Top-N Accuracy (ACC1/5, ACC1/10), Pearson Correlation Coefficient (PCC), and Spearman's Rank Correlation Coefficient (SRCC). For example, on GAIC, their ACC1/5 is 66.0% and ACC1/10 is 83.0%, with a PCC of 0.806 and SRCC of 0.787.
  • Superior Subjective Preference: On their own dataset, the method received the highest average number of likes from human subjects (87.57), with a low standard deviation (4.88), indicating strong user acceptance.
  • Visual Explainability: The predicted 10-layer score maps provide clear visual explanations for why a crop is good or bad, demonstrating the model's ability to distinguish attractive elements from distractions (e.g., a palace vs. a crowd in Figure 9).

Limitations and Future Work

  • Saliency Detection Dependency: The method relies on an external saliency detection method (Hou et al. [7]). While this method is effective, its limitations (e.g., misclassifying distractions as salient) can impact the generated score maps. The paper acknowledges that distractions can sometimes be salient enough to be distracting.
  • Computational Cost: While the paper mentions a sliding window for crop proposal and a CNN for evaluation, the overall computational cost for generating and evaluating many crops might still be a consideration for real-time applications on resource-constrained devices.
  • Future Work: The paper implicitly suggests that further improvements could come from more advanced saliency detection or by exploring other aesthetic factors beyond saliency and composition. The explainability offered by the 10-layer score map also opens avenues for user-interactive cropping tools.

Summary updated

BibTeX
@inproceedings{cheng2022re,
  author = {Yang Cheng and Qian Lin and Jan P. Allebach},
  title = {{Re-Compose the Image by Evaluating the Crop on More Than Just a Score}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure