SoftCollage: A Differentiable Probabilistic Tree Generator for Image Collage

Jiahao Yu, Li Chen, Mingrui Zhang, Mading LiCVPR 2022 · OpenReview

Abstract

Image collage task aims to create an informative and visual-aesthetic visual summarization for an image collection. While several recent works exploit tree-based algorithm to preserve image content better, all of them resort to hand-crafted adjustment rules to optimize the collage tree structure, leading to the failure of fully exploring the structure space of collage tree. Our key idea is to soften the discrete tree structure space into a continuous probability space. We propose SoftCollage, a novel method that employs a neural-based differentiable probabilistic tree generator to produce the probability distribution of correlation-preserving collage tree conditioned on deep image feature, aspect ratio and canvas size. The differentiable characteristic allows us to formulate the tree-based collage generation as a differentiable process and directly exploit gradient to optimize the collage layout in the level of probability space in an end-to-end manner. To facilitate image collage research, we propose AIC, a large-scale public-available annotated dataset for image collage evaluation. Extensive experiments on the introduced dataset demonstrate the superior performance of the proposed method. Data and codes are available at https://github.com/ChineseYjh/SoftCollage.

Introduction

Creating a high-quality image collage—one that is informative, aesthetically balanced, and preserves the salient parts of every image—is a complex optimization task. Modern state-of-the-art methods typically adopt a tree-based approach, where a canvas is recursively partitioned (horizontally or vertically) to create slots for images.

However, optimizing this tree structure has historically been a bottleneck. Because decisions like choosing a horizontal versus vertical cut are discrete, researchers have traditionally relied on hand-crafted heuristic rules or greedy algorithms. These methods are limited in their ability to explore the vast space of possible tree structures and cannot be optimized end-to-end using modern gradient-based deep learning techniques.

Methodology: Softening the Tree

To overcome these limitations, the authors introduce SoftCollage, which transforms discrete structural decisions into a Probabilistic Collage Tree (PCtree).

Overview of the SoftCollage framework compared to conventional tree-based methods.

Probabilistic Tree Generation

Instead of selecting a fixed cut type, SoftCollage predicts a probability distribution for structural choices. This "softening" turns a discrete search space into a continuous one, allowing for gradient-based navigation. The architecture includes:

  • Feature Extractor: A ResNet-50 backbone extracts semantic features, while an "InfoEmbed" module incorporates metadata such as image aspect ratios and target canvas dimensions.
  • Fusion Module: A self-attentive module merges features from child nodes to represent parent "boxes" within the tree.
  • Node & Edge Classifiers: These components predict the probabilities of cut types (Horizontal/Vertical) and child arrangements (Left/Right).

Differentiable Optimization

By modeling the tree as a distribution, the framework can utilize policy gradient techniques to optimize the generator. A "Criterion Loss" is defined to reward the model for minimizing blank space, preserving aspect ratios, and grouping semantically similar images together. This allows the layout to evolve toward a global optimum via backpropagation.

The AIC Dataset

A significant contribution of this work is the introduction of the Annotated Image Collection (AIC) dataset, addressing the lack of standardized evaluation benchmarks.

  • Scale: Derived from the DUTS saliency dataset and ImageNet, AIC includes 3,402 images across 72 categories and 11 themes.
  • Annotations: Each image is labeled with category, theme, and saliency masks, enabling quantitative measurement of content preservation and semantic grouping.

Experimental Results

SoftCollage was evaluated against commercial tools and the previous state-of-the-art (VSM).

Qualitative comparison between VSM and SoftCollage, highlighting improved semantic grouping and layout.

Quantitative and Qualitative Performance

SoftCollage demonstrated clear superiority in several key metrics:

  • Ratio Preservation: Achieved a score of 1.086, significantly better than SHP's 1.522.
  • Correlation Preservation: Scored 0.284 compared to SHP's 0.376, indicating better semantic grouping (as seen in the artichoke and orange examples in Figure 2).
  • Human Evaluation: In a 5-scale aesthetic study, SoftCollage was rated "Excellent" by 34.2% of participants, outperforming VSM (29.2%). In side-by-side win-rate tests, it surpassed VSM by a margin of 20.7%.

Conclusion

SoftCollage successfully bridges the gap between discrete tree-based layouts and differentiable deep learning. By treating tree structures as probability distributions, it enables the automated discovery of optimized, content-aware collages. While current implementations face GPU memory constraints as image collections grow, future work focusing on lightweight model designs promises to make this approach more scalable.

Summary updated

Abstract

Image collage task aims to create an informative and visual-aesthetic visual summarization for an image collection. While several recent works exploit tree-based algorithm to preserve image content better, all of them resort to hand-crafted adjustment rules to optimize the collage tree structure, leading to the failure of fully exploring the structure space of collage tree. Our key idea is to soften the discrete tree structure space into a continuous probability space. We propose SoftCollage, a novel method that employs a neural-based differentiable probabilistic tree generator to produce the probability distribution of correlation-preserving collage tree conditioned on deep image feature, aspect ratio and canvas size. The differentiable characteristic allows us to formulate the tree-based collage generation as a differentiable process and directly exploit gradient to optimize the collage layout in the level of probability space in an end-to-end manner. To facilitate image collage research, we propose AIC, a large-scale public-available annotated dataset for image collage evaluation. Extensive experiments on the introduced dataset demonstrate the superior performance of the proposed method. Data and codes are available at https://github.com/ChineseYjh/SoftCollage.
BibTeX
@inproceedings{yu2022softcollage,
  author = {Jiahao Yu and Li Chen and Mingrui Zhang and Mading Li},
  title = {{SoftCollage: A Differentiable Probabilistic Tree Generator for Image Collage}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=cd9eN5riti},
}
Figure