Spatial-Semantic Collaborative Cropping for User Generated Content

Yukun Su, Yiwen Cao, Jingliang Deng, Fengyun Rao, Qingyao Wu

Introduction

Image cropping is a fundamental task in computer vision, essential for optimizing visual presentation across diverse digital platforms. While traditional research has focused on "iconic" photography—images with clear, single subjects and clean backgrounds—the rise of social media and mobile platforms has shifted the focus toward User Generated Content (UGC). UGC is inherently more complex, often featuring multiple subjects, chaotic backgrounds, and critical text overlays like headlines or subtitles.

To address these challenges, this paper introduces the Spatial-Semantic Collaborative cropping network (S2CNet). S2CNet is designed to model the intricate relationships between multiple objects and their backgrounds to ensure "content integrity"—the preservation of essential visual information—while maintaining high aesthetic standards.

The Complexity of User-Generated Content

Existing cropping tools frequently fail when applied to UGC for three primary reasons:

  1. Neglect of Content Integrity: Aesthetic-driven models may produce a visually pleasing composition that inadvertently truncates a person's face or removes vital text.
  2. Contextual Blindness: Many models treat objects in isolation, failing to understand how different elements in a frame relate to one another.
  3. Rigid Constraints: Models often struggle to adapt to the specific, fixed aspect ratios required by modern mobile applications and web interfaces.

S2CNet addresses these issues by treating the image as a graph of interacting "visual genes," allowing the system to evaluate how every element contributes to the overall message and composition of the frame.

S2CNet: Modeling Spatial-Semantic Interactions

The architecture of S2CNet is built upon the concept of Visual Gene Mining. Using a pre-trained object detector (Faster R-CNN), the model identifies potential Regions of Interest (RoIs) within an image. These RoIs serve as the nodes in an Adaptive Attention Graph (AAG), which acts as the core reasoning engine of the system.

The AAG calculates two distinct types of relationships, or edges, between these nodes:

  • Semantic Edges: These measure the visual similarity and thematic connection between objects.
  • Spatial Edges: These define the topological relationship between objects.

A key innovation in S2CNet is the Distance Embedding (DisEmb) strategy. The authors observed that simply ignoring distant objects (Distance Dropping) reduced the model's ability to understand global context. DisEmb allows the model to learn that even far-apart objects—such as two people interacting across a room—can be aesthetically and semantically linked, ensuring they are both preserved in the final crop.

To maintain high performance, the authors modified standard Graph Convolutional Networks (GCNs) into a more efficient "Graph-Aware Attention" module. This allows for differentiable message passing, centralizing information into crop candidates to predict their aesthetic and integrity scores with minimal computational overhead.

UGCrop5K: A New Benchmark for Content Integrity

Data scarcity has historically hindered progress in UGC cropping. To solve this, the authors introduced UGCrop5K, the largest densely labeled dataset of its kind.

  • Scale: 5,000 images sourced from platforms like YouTube and Bilibili.
  • Annotations: 450,000 high-quality annotated candidate crops.
  • Diversity: The dataset covers a wide range of categories, including gaming, vlogs, news, and sports, providing a robust testing ground for content integrity.

Experimental Results and Efficiency

S2CNet was evaluated against existing benchmarks (GAICv1 and GAICv2) and the new UGCrop5K dataset. It consistently achieved State-of-the-Art (SOTA) performance in terms of accuracy (ACC) and Spearman’s Rank Correlation Coefficient (SRCC), outperforming previous models like TransView and HCIC.

Beyond quantitative metrics, the model excels in practical application:

  • Speed: S2CNet achieves a throughput of 162.8 FPS on a single NVIDIA RTX 2080Ti, making it suitable for real-time server-side processing.
  • User Preference: In subjective studies, human volunteers preferred S2CNet’s crops significantly more than those of competing algorithms. For instance, in 16:9 aspect ratio tests, S2CNet achieved a 52% preference rate, more than double its closest competitor.

Conclusion

S2CNet represents a significant advancement in automated image cropping by prioritizing the relationship between spatial topology and semantic content. By introducing the UGCrop5K dataset and the DisEmb strategy, the authors have provided both a framework and a foundation for better understanding content integrity in the messy, multi-subject world of UGC. Future work will look toward extending this spatial-semantic graph approach to temporal video cropping, where the optimal crop must dynamically track subjects over time.

Summary updated

BibTeX
@inproceedings{su2024spatial,
  author = {Yukun Su and Yiwen Cao and Jingliang Deng and Fengyun Rao and Qingyao Wu},
  title = {{Spatial-Semantic Collaborative Cropping for User Generated Content}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i5.28303},
}
Figure