TL;DR
Collaposer is an AI-powered creativity support tool that automates the tedious process of preparing object-level visual assets (cutouts) from large photo collections for digital collage storytelling. By coupling multi-model computer vision with an LLM, it extracts, selects, and hierarchically organizes assets based on a user's text narrative, allowing creators to seamlessly transition from abstract concepts to static or animated compositions.
What Problem Does This Paper Address?
Digital collage is a powerful, expressive art form, but creating one is notoriously labor-intensive. Creators must manually sift through massive photo collections, execute precise and tedious image cutouts (segmentation), and organize hundreds of fragments before composition can even begin. This mechanical overhead disrupts creative momentum.
Furthermore, traditional asset management systems operate at the level of whole images and rely on rigid keyword searches. This makes it difficult for creators to translate loose story ideas into effective search queries or discover contextually relevant assets that aren't explicitly named. Collaposer addresses these bottlenecks by automating object-level extraction, semantic asset selection, and hierarchical organization.
Methods
Collaposer introduces a story-driven asset preparation pipeline divided into three distinct stages:
- Photo Collection Preprocessing (Stage I): The system processes raw images using a suite of vision models. It utilizes the Recognize Anything Model (RAM) for zero-shot image tagging, Grounding DINO for open-set object detection bounding boxes, and the Segment Anything Model (SAM) to generate clean, object-level RGBA cutouts.
- Element Selection and Clustering (Stage II): Using GPT-4o, the pipeline analyzes the user's text narrative against the extracted asset tags. Crucially, it performs divergent selection, identifying both "central labels" (explicitly mentioned in the text) and "related labels" (contextually implied assets to enrich the scene). The LLM then classifies these assets into functional roles (characters, backgrounds, accessories) and organizes them into a nested semantic hierarchy. For character assets, Sapiens is used to perform 2D part segmentation and keypoint estimation, automatically defining rotation centers for future animation.
- Asset Presentation (Stage III): Assets are displayed in a user interface featuring a linked tree view and canvas layout. Elements are adaptively sized based on a composite selection score that balances content diversity, story consistency (via CLIP embedding similarities), and image resolution.
Datasets
The system was evaluated using a fixed photo set of 116 images preprocessed into 2,427 distinct visual elements. The images were randomly sampled from public-domain historical and scientific digital archives:
- The NASA Archives
- The Olympics Archive
- The Library of Congress
No new machine learning benchmarks were introduced; these datasets served as the testing ground for the user evaluation and application scenarios.
Key Insights and Ideas
- Semantic Inference Sparks Serendipity: Traditional search limits creators to what they can explicitly name. By leveraging an LLM to infer contextually relevant assets (e.g., suggesting "clouds" and "trees" for a landscape story), Collaposer expands the creative space and provides unexpected visual inspiration.
- Functional Role Classification Reduces Cognitive Load: Displaying thousands of cutouts in a flat grid is overwhelming. Sorting assets into structural roles (characters, backgrounds, accessories) mirrors the natural workflow of artists, making asset evaluation significantly more intuitive.
- Pre-Rigging for Animation: By integrating part-level parsing and keypoint estimation directly into the asset preparation phase, the tool bridges the gap between static graphic design and dynamic animation tools like Adobe After Effects.
Key Results
The authors conducted a within-subjects user study ($N=12$) benchmarking Collaposer against two ablated baselines: Ablated-Select (no semantic inference/related labels) and Ablated-Present (no semantic clustering or adaptive sizing).
- 100% One-Pass Success: Under a strict 15-minute task budget, all participants using Collaposer achieved a satisfactory asset set on their very first prompt submission, requiring zero re-prompting iterations.
- Superior Usability and Alignment: Quantitative 7-point Likert scale ratings showed that Collaposer significantly outperformed Ablated-Select in prompt alignment and visual expectation, and significantly beat Ablated-Present in ease of element identification.
- Downstream Integration: The exported structured JSON files and layered cutouts were successfully imported into commercial animation software via scripts, enabling rapid production of complex, articulated stop-motion animations.
Limitations and Future Work
- Text-Based Constraints: Relying solely on text prompts can occasionally cause misalignments between a user's mental image and the system's output. Incorporating multimodal inputs (like sketch-based queries) is a suggested next step.
- Lack of Transparency: Users occasionally found the LLM's semantic inferences confusing, pointing to a need for explainable AI features where the system explains why an asset was recommended.
- Single-Pass Workflow: The current pipeline operates linearly. Future work aims to support continuous, iterative refinement loops, allowing users to dynamically rename, regroup, or re-segment assets on the fly without restarting the pipeline.
Summary updated