Layered Depth Ordering for Scene Synthesis and Decomposition

This direction is split into other directions: depth ordering ambiguity. Its last version stays readable and citable here.

Version 1 of 176 papers85 conceptsfrozen 2026-10-04

Most computer vision models treat an image as a flat grid of pixels. While this works for simple classification, it fails in the "messy" real world where objects overlap, obscure one another, and exist in a specific front-to-back sequence. To solve this, a growing body of research proposes moving away from flat representations toward Layered Depth Ordering—a "2.5D" approach that treats a scene as a stack of discrete, ordered layers.

The Problem of the Flat Grid

When a model generates or analyzes an image as a single 2D plane, it lacks an inherent understanding of the z-axis. This leads to "ghosting" artifacts, where background textures bleed into foreground objects, or "limb-tangling" in multi-person animations. As Bo Cheng and colleagues note in HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation, standard models often rely on global prompts to guess which object is in front, leading to physically impossible arrangements in complex scenes.

Without explicit depth reasoning, a system cannot distinguish between an object ending and an object being hidden. This "spatial ambiguity" is why Zehan Wang and the SpatialHand team argue that generative models struggle to place objects realistically relative to existing scene elements.

The Painter’s Algorithm as a Framework

The core intuition behind this research direction is the Painter’s Algorithm: the idea that a scene should be constructed by painting the background first and then layering foreground objects on top.

In A Layer-Based Sequential Framework for Scene Generation with GANs, Mehmet Ozgur Turkoglu and his team demonstrate that by adding objects one by one, a model can handle occlusions more naturally and provide finer user control. This logic is extended to modern diffusion models by Dong Liang in VODiff, which treats image synthesis as a stack of layers with a specific "visibility order" (or z-index). By assigning each object its own "lane," models like HiCo ensure that lighting and boundaries remain consistent even in crowded layouts.

Decomposing the Stack

If synthesis is about building the stack, decomposition is about unweaving it. Traditional segmentation often fails at shared boundaries because it doesn't know which object "owns" the edge. Xuemei Cheng and Thomas R. Tsao propose a Border Ownership (BOWN) framework that mimics biological vision to organize scenes into "figures" and "backgrounds."

A key breakthrough in this area is the move from physical depth (meters) to ordinal depth (stacking order). In Illustrator's Depth: Monocular Layer Index Prediction for Image Decomposition, Nissim Maruani and Yifan Wang show that predicting a "layer index" is often more useful for creative tasks than predicting actual distance. This allows for unsupervised decomposition, where models like the one proposed by Tom Monnier in Unsupervised Layered Image Decomposition into Object Prototypes learn to extract "sprites"—prototypical objects with their own transparency masks—simply by watching how they move and overlap.

Resolving Overlaps with Graphs and Ranking

When multiple object hypotheses exist for the same pixels, how does a model decide which one is on top? Earlier methods used rigid heuristics, but Yibo Yang and colleagues introduced SOGNet, which uses a Scene Overlap Graph to treat overlap resolution as a learnable relational problem.

Similarly, Xu Liu introduced a Spatial Ranking Module in An End-To-End Network for Panoptic Segmentation. Instead of relying on simple confidence scores—which might delete a valid but "uncertain" foreground object—these modules learn the logical arrangement of objects (e.g., a tie is usually in front of a shirt).

Amodal Vision: Seeing the Unseen

The ultimate test of layered depth ordering is amodal reasoning: the ability to perceive the full shape of an object even when it is partially obscured. Maximilian Luz and Rohit Mohan explore this in Amodal Optical Flow, where they predict motion for entire objects, including their hidden regions.

By representing a scene as a hierarchy of depth-ordered motion layers, models can propagate information from visible parts to occluded ones. As Yuka Kihara demonstrates in In the Shadows, Shape Priors Shine, combining depth awareness with shape priors allows a system to "hallucinate" the boundaries of a background object that are completely hidden from view, effectively reconstructing the 2.5D reality of the scene.

Papers in this version

Showing 60 of 76 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_layered_depth_ordering_for_scene_synthes_5d2248a7,
  title = {Layered Depth Ordering for Scene Synthesis and Decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/layered-depth-ordering-for-scene-synthesis-and-decomposition/rdr_5d2248a781524e7c818d3ccd92fd3de3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_layered_depth_ordering_for_scene_synthes_5d2248a7_v1,
  title = {Layered Depth Ordering for Scene Synthesis and Decomposition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/layered-depth-ordering-for-scene-synthesis-and-decomposition/rdr_5d2248a781524e7c818d3ccd92fd3de3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}