When we look at a two-dimensional image, our brains instantly reconstruct a three-dimensional environment. We know that a coffee cup sitting in front of a laptop partially blocks our view of the keyboard, and we understand that the laptop continues to exist behind the cup.
For computer vision and generative models, however, this "z-axis" is entirely missing by default. Standard deep learning architectures operate on flat grids of pixels. When multiple objects occupy the same spatial region—whether during image synthesis, object editing, or scene parsing—models face depth ordering ambiguity. Without an explicit mechanism to determine which object is in front, models produce severe visual artifacts: objects unnaturally blend into a "purple mess," foreground elements are overwritten during sequential edits, or newly inserted objects are simply stamped on top of background structures like trees or buildings.
To overcome this limitation, a prominent research direction focuses on explicitly modeling spatial depth hierarchies and occlusion relationships. By shifting from flat pixel grids to layered, depth-aware representations, these techniques allow models to reason about what is visible, what is hidden, and how objects are ordered from front to back.
The Challenge of the Missing Z-Axis
To understand why standard models fail, consider the task of layout-to-image generation. When a user provides overlapping bounding boxes or masks, a standard text-to-image diffusion model often struggles to resolve spatial conflicts (Marianna Ohanyan et al.). Because the model lacks an inherent understanding of the z-axis, it relies entirely on global text prompts to guess the layout hierarchy (Bo Cheng). This lack of explicit depth or occlusion handling causes collision artifacts and unnatural blending when objects are moved or generated in overlapping positions (Omri Avrahami et al.).
A similar failure mode occurs in image composition and editing. Traditional automated blending systems and object insertion models frequently lack the spatial reasoning required to manage depth boundaries (Haochen Xue et al.). Instead of realistically embedding a new object behind an existing foreground element, they simply overlay the new pixels on top (Donghoon Lee 0004 et al.; Chaohua Shi et al.). Furthermore, during sequential image editing, models suffer from "memory loss"; they struggle to maintain background consistency because they cannot determine whether a new edit should overwrite or slip behind existing layers (Daneul Kim et al.).
Decomposing Scenes into Hierarchical Layers
The core intuition behind resolving this ambiguity is simple: treat the image not as a single flat canvas, but as a stack of discrete, ordered layers.
Early generative approaches adopted a digital painter’s metaphor to handle occlusions. By utilizing a sequential framework, a model can generate the background first and then add foreground objects one by one (Mehmet Ozgur Turkoglu et al.). This sequential addition prevents the "ghosting" or merged-shape artifacts that occur when a network tries to resolve multiple overlapping boundaries simultaneously. A similar layer-based assembly process allows models to infer the correct sequence of additions even in highly specific domains, such as tracking how ingredients are layered on top of one another (Dim P. Papadopoulos et al.).
Flat Generation: [ Background + Foreground Objects Generated Simultaneously ]
│
▼ (Ambiguous boundaries, ghosting)
Layered Generation: [ Background Layer ] ──► [ Midground Layer ] ──► [ Foreground Layer ]
│
▼ (Clear occlusion, crisp boundaries)
Modern diffusion models scale this intuition by giving different spatial elements their own independent processing paths. For example, the architecture proposed by Bo Cheng in HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation uses a multi-branch structure to model background and foreground elements independently before fusing them. By separating the layers during the generation process, the model eliminates the risk of background elements bleeding into the foreground, preventing physically impossible renderings where spatial layering is ambiguous (Bo Cheng et al.).
Explicit Depth Injection and 3D Conditioning
While layering provides a structural framework, models still need a mechanism to decide which layer belongs in front. This requires injecting explicit depth signals or 3D geometric constraints into 2D models.
One way to achieve this is through specialized training objectives that decouple depth from texture. Jonghyun Lee et al. introduced Depth Disentanglement Training (DDT), which forces a diffusion model to learn an explicit "sense of depth." By training the model to separate localized region semantics from their corresponding z-axis positions, the framework can compose complex scenes where objects correctly occlude one another based on reference depth maps.
For even finer control, researchers have begun decomposing 2D image manipulation into explicit 3D representations. Zehan Wang et al. developed a framework that breaks down object manipulation into 2D masks, depth maps, and latent orientation features. By explicitly reasoning about the 3D perspective, the model can reliably determine whether a newly inserted object should appear in front of or behind other components, eliminating spatial placement errors. This explicit modeling of front-to-back spatial relationships is similarly crucial for rendering realistic focus boundaries (Parikshit Sakurikar et al.) and maintaining tracking robustness when similar objects overlap from the camera's perspective (Niklas Gard et al.).
Amodal Perception: Seeing the Hidden Whole
Resolving depth ordering is not just a generative challenge; it is equally vital for scene understanding. When objects interact closely—such as hands in dense interaction (Zhengdi Yu et al.), moving parts of an object (Qingzhe Gao et al.), or crossing paths in video frames (Hsu-kuang Chiu)—standard 2D segmentation models fail because they only track visible pixels (Buyu Liu).
To truly parse a scene, models must employ amodal segmentation—the ability to predict both the visible and the occluded parts of an object (Semantic Amodal Segmentation). By formulating scene parsing with explicit occlusion ordering constraints, models can reconstruct the full extent of partially hidden entities and understand how they are layered within the environment (Joseph Tighe et al.).
Visible Segmentation: [ Traces only the exposed parts of an object ]
Amodal Segmentation: [ Hallucinates the hidden structure + Infers layer order ]
However, simply predicting that an object exists in an occluded region does not inherently establish its exact depth hierarchy (Pulak Purkait et al.). To resolve this final layer of ambiguity, Jiayang Ao et al. introduced a Cumulative Occlusion Learning strategy within a diffusion framework. Their model processes the image layer-by-layer, using the masks of foreground objects as explicit spatial context to infer the shapes of the hidden layers underneath. This allows the network to exploit subtle visual cues like shadows and T-junctions, ensuring that even when visual evidence is highly ambiguous, the predicted spatial hierarchy remains physically plausible.
Key Foundational Work
Vehicle Segmentation and Tracking from a Low-Angle Off-Axis Camera (2005) — contributed 1 concept elements to this direction.
Switching Hypothesized Measurements: A Dynamic Model with Applications to Occlusion Adaptive Joint Tracking (2003) — contributed 1 concept elements to this direction.
Guiding Model Search Using Segmentation (2005) — contributed 1 concept elements to this direction.