Decomposing 3D Scenes into Editable Semantic Parts

This direction is split into other directions: modular scene decomposition. Its last version stays readable and citable here.

Version 1 of 1230 papers281 conceptsfrozen 2026-10-04

Traditional 3D generative models often treat an object or a scene as a monolithic "blob"—a single mesh or a continuous volume of density. While this approach can produce visually impressive results, it creates a "structural nightmare" for practical use. If you want to move a chair in a generated room or animate a character’s arm, a monolithic representation fails because the underlying geometry is fused together.

A growing body of research is shifting toward compositional 3D generation, where scenes and objects are synthesized as assemblies of discrete, semantically meaningful parts. By decoupling the representation into independent entities, these methods achieve a level of editability and structural logic that monolithic models cannot match.

The Problem with Monolithic Volumes

In early text-to-3D work, models optimized a global Neural Radiance Field (NeRF) or mesh. As Xi Tian and colleagues point out in ShapeScaffolder, this often results in "missing legs" or "blurry textures" because the model lacks an internal understanding of an object's components. When the entire object is one mathematical function, modifying a specific attribute—like the curve of a wooden armrest—inevitably leaks into the rest of the geometry.

To solve this, researchers are moving toward semantic decomposition. The intuition is simple: if the AI knows that a "chair" consists of a "back," "seat," and "four legs," it can generate and optimize those parts independently.

Layouts as the Architectural Scaffold

How does a model know where to place these parts without a global reference? The emerging solution is to use Large Language Models (LLMs) as "architects."

In GALA3D, Xiaoyu Zhou uses LLMs to plan a structured 3D layout of bounding boxes before any geometry is generated. Similarly, Yiqi Lin and the team behind LI3D use structured layouts as a "universal interface" that decouples semantic reasoning from visual rendering.

By factoring the problem into a semantic layout stage followed by a geometric synthesis stage, models like SceneFactor allow users to move a box in the layout to move the corresponding object in the 3D scene. This "local-to-global" optimization ensures that each object maintains its integrity even as the overall scene composition changes.

From Implicit Blobs to Explicit Gaussians

The choice of 3D representation is critical for modularity. While NeRFs are continuous and hard to "cut," 3D Gaussian Splatting (3DGS) offers an explicit, point-based representation that is naturally suited for composition.

Chenfeng Xu and the authors of CompGS leverage this by representing scenes as collections of entity-aware Gaussian clusters. Because Gaussians are discrete primitives, they can be scaled, rotated, or deleted without affecting the surrounding space. This is further explored in DreamScape, where Xuening Yuan uses a structured Gaussian guide to prevent "textual guidance collapse"—a common failure where the model confuses the attributes of different objects (e.g., making the table blue because the chair is blue).

Part-Level Generation for Production

For professional workflows, even "object-level" decomposition isn't enough. Assets need to be "part-aware" for animation.

Minghao Chen and Iro Laina introduced PartGen, which uses multi-view diffusion to "hallucinate" the complete geometry of occluded parts. This ensures that when a part is moved, there isn't a "hole" behind it. In the realm of character generation, Yuze He and the StdGEN team automatically separate body, hair, and clothing into distinct layers, mimicking how human artists build 3D characters.

This modularity extends even to complex natural structures. Zhihao Liu demonstrates in Neural Hierarchical Decomposition that plants can be reconstructed as a hierarchy of 3D boxes. This allows for botanical-based parametric modeling where individual branches can be simulated or edited independently.

Decoupling Reconstruction from Editing

Finally, the most flexible systems are those that decouple the 3D representation from the editing process entirely. Chat-Edit-3D achieves this by using a "Hash-Atlas" representation, allowing 2D "visual experts" (like Stable Diffusion) to perform edits that are then projected back into 3D.

Similarly, Verica Lazova and the Control-NeRF team use editable feature volumes. Instead of baking the scene into the weights of a neural network, they store data in a grid that can be resampled. This allows for "copy-paste" operations in 3D space—moving or duplicating objects by simply shifting their feature voxels.

By treating 3D space as a collection of independent, semantically labeled parts rather than a single continuous field, these researchers are bridging the gap between "AI-generated blobs" and "production-ready assets."

Papers in this version

Showing 60 of 230 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_decomposing_3d_scenes_into_editable_sema_e3b8613d,
  title = {Decomposing 3D Scenes into Editable Semantic Parts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decomposing-3d-scenes-into-editable-semantic-parts/rdr_e3b8613d9928432cb5b072ea574ae380},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_decomposing_3d_scenes_into_editable_sema_e3b8613d_v1,
  title = {Decomposing 3D Scenes into Editable Semantic Parts},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/decomposing-3d-scenes-into-editable-semantic-parts/rdr_e3b8613d9928432cb5b072ea574ae380/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}