Layered and Part-Based 3D Representations

38 concepts36 papers1 proposal

When deep learning models reconstruct or generate 3D data, their default tendency is to produce a single, continuous surface. Whether optimizing a neural radiance field, a 3D Gaussian splat, or a traditional mesh, standard loss functions reward the model purely for matching the overall visual appearance from training viewpoints. This results in a monolithic representation: a single "watertight" shell where distinct physical entities are fused together.

While a monolithic mesh might look convincing from a fixed angle, it suffers from severe geometric entanglement. In the real world, objects are modular assemblies, scenes are collections of independent entities, and humans wear clothes that slide over their skin. When a model bakes these distinct layers into a single surface, it creates a topological nightmare. If you try to animate a character's arm, the torso fabric stretches unnaturally. If you remove a chair from a generated room, it leaves a gaping hole in the floor rather than revealing the surface underneath.

To build truly interactive digital worlds, researchers are moving away from monolithic shells toward layered and part-based 3D representations.

The Problem of Geometric Entanglement

The limitations of monolithic modeling are most apparent in digital humans. Early reconstruction and capture methods, as noted by Tao Yu et al. in SimulCap : Single-View Human Performance Capture With Cloth Simulation, treat the body and clothing as a single fused layer. This prevents the realistic simulation of fabric dynamics and makes downstream tasks like virtual try-on impossible. Similarly, Zhaoqi Su et al. observe in MulayCap: Multi-Layer Human Performance Capture Using a Monocular Video Camera that such "painted-on" textures fail to model the physical interactions between garments and skin, leading to severe stretching artifacts during motion.

The problem is not unique to humans; it scales from individual objects up to entire environments:

  • Human Avatars: Fusing skin, clothing, and hair into one mesh means garments cannot be swapped or simulated independently.
  • Rigid Objects: Fusing a chair's legs to its seat means that modifying or scaling one component distort the others.
  • Scenes: Fusing multiple interacting objects means the scene cannot be rearranged; removing an object creates a geometric void.

Decomposing the Human Body into Layered Assets

To solve geometric entanglement in digital humans, researchers decouple the stable, underlying human structure from the highly variable outer layers (clothing and hair).

An early milestone in this direction was achieved by Bharat Lal Bhatnagar et al. with Multi-Garment Net: Learning to Dress 3D People From Images, which reconstructs the body and garments as separate, layered meshes. By leveraging parametric body models like SMPL-X (detailed by Georgios Pavlakos et al. in Expressive Body Capture: 3D Hands, Face, and Body From a Single Image), modern frameworks can anchor independent clothing layers to a shared skeletal rig.

Monolithic Approach:
[ Body + Shirt + Pants ] ---> Optimized as a Single Fused Mesh (No Under-clothing Geometry)

Layered Approach:
[ Parametric Body Model ] + [ Independent Shirt Layer ] + [ Independent Pants Layer ]
                                       |
                                       v
                        Gradients Optimize Each Layer Natively

In text-to-3D generation, frameworks like TELA (developed by Junting Dong et al.) and DAGSM (developed by Jingyu Zhuang et al.) use stratified rendering and 2D diffusion priors to optimize separate geometry layers simultaneously.

Single-author work by Hui En Pang, Disco4D: Disentangled 4D Human Generation and Animation from a Single Image, demonstrates how combining the structural stability of a parametric body model with the flexible detailing of 3D Gaussian Splatting for garments allows the system to "inpaint" the occluded skin beneath the clothes. This explicit separation ensures that when the character moves, the garments deform independently without tearing or clipping the underlying body mesh.

Part-Aware Object Generation and the Resolution Bottleneck

The monolithic problem extends directly to rigid objects. When a generative model treats a complex object—like a chair or a car—as an indivisible unit, it lacks structural awareness. Nicolas Talabot et al. point out in PartSDF: Part-Based Implicit Neural Representation for Composite 3D Shape Parametrization and Optimization that modifying a single feature in a standard implicit model often introduces unwanted distortions across the entire shape (e.g., stretching a car chassis might accidentally turn its wheels oval).

Furthermore, monolithic models suffer from a resolution bottleneck. If a single voxel grid or coordinate space represents the entire object, small but crucial components (like chair legs or door handles) receive only a tiny fraction of the total resolution budget.

To overcome this, part-aware architectures decompose objects into discrete, semantically meaningful sub-structures. Dan Xu addresses the resolution bottleneck in FullPart: Generating each 3D Part at Full Resolution by assigning every individual part its own dedicated, high-resolution voxel grid, maintaining global coherence through implicit layout diffusion.

Similarly, Jiaxiang Tang et al. introduce a dual volume packing strategy in Efficient Part-level 3D Object Generation via Dual Volume Packing to generate an arbitrary number of editable parts efficiently. By isolating each component's geometry, these models ensure that local edits remain local, preventing geometric bleeding and enabling clean, part-level articulation.

Dissecting Complex Scenes into Modular Assets

At the largest scale, text-to-3D scene generation frequently suffers from "blended" geometry, where multiple interacting objects are baked into a single inseparable mesh. As Zizheng Yan et al. explain in DreamDissector: Learning Disentangled Text-to-3D Generation from 2D Diffusion Priors, this lack of structural decomposition makes individual object animation or replacement impossible.

The solution lies in compositional scene generation, which enforces a strict mapping between discrete semantic concepts and independent 3D assets. For example, Ling Wang et al. propose SceneTransporter: Optimal Transport-Guided Compositional Latent Diffusion for Single-Image Structured 3D Scene Generation, which uses optimal transport to map image patches directly to distinct 3D parts, eliminating fragmented or blended geometry.

In hierarchical frameworks like HiScene by Bangbang Yang et al., the generation pipeline incorporates video-diffusion-based amodal completion. When generating a room, the model doesn't just render the visible surfaces; it explicitly infers and completes the hidden, occluded parts of every object. Consequently, if a user removes a sofa from the scene, the floor and wall behind it are already fully realized assets rather than empty voids.

Why It Works: Independent Coordinate Spaces

The fundamental reason why layered and part-based approaches succeed where monolithic models fail comes down to independent coordinate spaces and localized optimization.

In a monolithic model, every point in space is evaluated against a single global function. This creates a high degree of interdependence: changing the geometry of a jacket alters the gradients flowing through the underlying torso. By breaking the asset down into separate layers or parts, each component is granted its own local coordinate system, resolution budget, and deformation field.

This modularity provides three distinct advantages:

  1. Amodal Completion: It forces the network to predict geometry that is hidden from sight (like the skin under a shirt or the wall behind a cupboard), making the asset truly three-dimensional and interactive.
  2. Gradient Isolation: Optimization gradients for one part do not interfere with another, preventing the "blurring" or "gluing" artifacts common in end-to-end single-mesh pipelines.
  3. Downstream Compatibility: The resulting assets natively match the modular pipelines used in traditional computer graphics, physics engines, and animation software, bridging the gap between deep-learning-based generation and practical utility.

Key Foundational Work

Multi-Garment Net: Learning to Dress 3D People from Images (2019) — contributed 1 concept elements to this direction.
SimulCap : Single-View Human Performance Capture With Cloth Simulation (2019) — contributed 1 concept elements to this direction.
3D Mesh Automatic Blanking and Split Technology in Urban Planning (2013) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_monolithic_surface_representation_3ae7a232,
  title = {monolithic surface representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/monolithic-surface-representation/rdr_3ae7a2324ab24ddba4de947399317c23},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_monolithic_surface_representation_3ae7a232_v1,
  title = {monolithic surface representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/monolithic-surface-representation/rdr_3ae7a2324ab24ddba4de947399317c23/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake