When we look at a photograph, our brains instantly resolve a complex 3D hierarchy: a pedestrian stands behind a lamp post but in front of a building. For traditional generative models, however, an image is often just a flat grid of pixels. This lack of spatial reasoning leads to the "sticker effect," where newly inserted objects appear to float on top of the scene rather than being integrated into its volume.
To solve this, researchers are moving toward a "2.5D" approach to image synthesis. By explicitly modeling relative depth and occlusion, these systems can realistically tuck objects behind foreground elements, ensuring that spatial relationships remain physically plausible.
The Layering Problem in 2D Synthesis
The fundamental challenge in image composition—whether you are inserting a car into a street scene or editing a bowl of fruit—is determining the correct visibility of overlapping pixels. Standard diffusion models and GANs often struggle with this because they lack an internal representation of depth layers.
As noted in Improving Editability in Image Generation with Layer-wise Memory, models frequently fail during sequential edits, such as trying to place a new object behind an existing one. Without an explicit understanding of layer order, the model may incorrectly overwrite foreground elements. This issue is echoed in the work of Liyao Jiang and colleagues on PixelMan, who point out that without explicit occlusion handling, frameworks struggle to position target objects behind existing scene elements, leading to visual inconsistencies.
From "Where" to "How Deep"
To move beyond simple overlays, researchers have begun splitting the synthesis task into specialized modules. A seminal approach by Donghoon Lee and the NVIDIA team in Context-aware Synthesis and Placement of Object Instances separates the decision of where an object belongs from what shape it should take. This allows the system to account for depth and occlusion, preventing artifacts where new objects are simply "pasted" over foreground structures like trees or buildings.
Similarly, Xiaojuan Qi and Vladlen Koltun introduced Semi-parametric Image Synthesis (SIMS), which uses a dedicated ordering mechanism. By retrieving real photographic segments and determining their front-to-back layering, SIMS ensures that pedestrians are correctly placed in front of buildings, rather than being merged into them.
Mechanisms for Resolving Occlusion
How do these models actually "decide" what is in front? Several distinct mechanisms have emerged:
- Relative Depth Sub-networks: In Learning Object Context for Novel-view Scene Layout Generation, Xiaotian Qiao uses a specialized sub-network to determine the relative depth ordering of individual object masks. This prevents "ghosting" and physically impossible overlaps when the camera view changes.
- Depth-Based Aggregation: For dynamic tasks like Panoptic Segmentation Forecasting, Colin Graber and Alex Schwing use predicted depth values to layer "things" (cars, people) over "stuff" (roads, buildings). This ensures that as objects move in a predicted future frame, they overlap correctly based on their distance from the camera.
- Monocular Depth Priors: Newer pipelines like Cut-and-Splat by Bram Vanherle leverage monocular depth estimation to identify horizontal surfaces and depth layers in 2D backgrounds. This allows synthetic 3D objects (Gaussian Splats) to be "inserted" into a specific depth slice of a 2D image.
Amodal Completion: Seeing the Unseen
A critical component of realistic occlusion is knowing what an object looks like behind the thing covering it. This is known as amodal completion. Jiayang Ao and Krista A. Ehinger addressed this in Open-World Amodal Appearance Completion. Their system identifies occluders by combining object detection with depth-ordering networks. By understanding which segments lie in front, the model can then use generative inpainting to reconstruct the hidden parts of the target object, making the final composition feel three-dimensional.
The Future of Layered Composition
The shift toward depth-aware synthesis is solving the "forgetfulness" and spatial chaos of early diffusion models. Frameworks like LLM Blueprint by Hanan Gani and Salman Khan now use automated layout planning to manage complex scenes. By treating image generation as an assembly of depth-sorted layers rather than a single-pass pixel prediction, these models can finally handle the intricate "hide-and-seek" of objects in the real world.