Abstract
To quickly synthesize complex scenes, digital artists often collage together visual elements from multiple sources. We propose a similar process to parse a scene, modeling it as a collage of warped, layered objects sampled from labeled reference images. Each object is related to the rest by support constraints. Scene parsing is achieved through analysis-by-synthesis: starting with a dataset of labeled exemplar scenes, we retrieve a dictionary of candidate object segments and combine them into a "scene collage" that explains the query image. This representation recovers object counts, support relationships, ordinal depth, and occluded content, enabling applications like image editing, random scene synthesis, and image-to-anaglyph generation.
Introduction
Traditional semantic segmentation treats images as flat 2D grids of pixel labels. While effective for identification, this approach ignores the 3D structure of the world where objects exist in layers and occlude one another. Background structures are often fragmented by foreground occluders, a reality that "flat" representations fail to capture. To bridge this gap, we introduce "Scene Collaging," a model that understands occlusion, ordinal depth, and support relationships (e.g., a car supported by a road).
The Scene Collage Model
Our approach is inspired by digital artists who create complex scenes by layering visual elements. We model a scene as a collection of segments retrieved from a large database of human-annotated images. Each segment is warped (translated and scaled) and assigned to a discrete depth layer.
To ensure structural coherence, we use a "scenegraph" to represent support constraints. For instance, a "tree" segment is validated by the presence of a "field" segment beneath it. This "common sense" check prevents physically impossible placements, such as a car floating in the sky.
Analysis-by-Synthesis via Scene Grammar
Parsing a query image involves finding the combination of segments that best explains its appearance. We employ a nonparametric scene grammar that explores the space of possible collages using three primary moves:
- Birth: Adding a new object to the scene.
- Death: Removing an existing object.
- Swap: Exchanging one object for another from the dictionary.
The system greedily optimizes the posterior probability of the scene, balancing visual fit with the structural rules learned from exemplar data. This allows the model to "fill in" content that is occluded, treating fragmented objects as single, continuous layers.
Applications and Results
The model was evaluated on the LabelMe Outdoor (LMO), SUN, and NYU RGBD v2 datasets. While its pixel-labeling accuracy (70.0% on LMO) was slightly lower than contemporary "flat" methods (~78%), it provided a much richer representation of the scene.
This generative capability enables several unique applications:
- Image Editing: Objects can be swapped or moved while automatically maintaining depth layers and handling occlusions.
- Image-to-Anaglyph: By shifting the inferred layers, the system can generate 3D effects from a single 2D image.
- Random Scene Synthesis: By taking a "random walk" through the scene grammar, the system can synthesize entirely new, plausible scenes.
Conclusion and Future Work
Scene Collaging moves computer vision from pure classification toward a generative understanding of 3D space. However, the current model assumes objects are flat polygons at specific depths, which struggles with surfaces that recede into the distance, like roads. Future work will address these geometric limitations and aim to reduce the computational cost, which currently requires significant time per image.
Summary updated