Standard neural scene representations, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS), are remarkably good at photorealistic reconstruction. However, they suffer from a fundamental "monolithic" flaw: they represent the world as a continuous soup of density and color. To a standard NeRF, there is no difference between a coffee mug and the table it sits on; they are simply regions of high density in a single, inseparable volume.
This lack of structural decomposition prevents robots from interacting with specific items, stops users from editing scenes, and hinders high-level reasoning. A new wave of research seeks to "un-render" these monolithic volumes into discrete, object-centric components.
From Monoliths to Assemblies
The core challenge is moving from a single global function to a compositional representation. Instead of training one large Multi-Layer Perceptron (MLP) to represent an entire room, researchers like Shikun Liu and Andrew J. Davison in vMAP propose representing every object as its own independent, "tiny" neural field. By optimizing these models in parallel, the scene becomes a collection of "watertight" entities rather than a single geometric shell.
This shift allows for what Karl Stelzner and colleagues call "un-rendering" in ObSuRF. By treating the scene as an assembly of parts, models can perform semantic segmentation and 3D reconstruction simultaneously. As noted in ANISE by Dmitry Petrov, disentangling a shape's structure from its geometry is the key to enabling intuitive part-level editing.
Lifting 2D Semantics into 3D Space
How does a model learn where one object ends and another begins without manual 3D labels? The dominant strategy is Feature Field Distillation.
Researchers leverage "foundation models" that already understand objects in 2D—such as the Segment Anything Model (SAM), CLIP, or DINO—and "lift" their knowledge into the 3D volume.
- Distilled Feature Fields (DFFs), introduced by Sosuke Kobayashi and Vincent Sitzmann, embed 2D semantic features directly into a 3D radiance field. This allows users to select or move objects using simple text queries.
- Total-Decom by Xiaoyang Lyu uses SAM to guide a mesh-based region-growing algorithm, enabling object isolation with just a few user clicks.
- FeatureNeRF distills 2D priors to enable zero-shot tasks like part segmentation across different object instances.
By supervising the 3D representation with these 2D semantic "teachers," the model learns to group points or Gaussians into coherent entities that match human concepts.
Motion as a Grouping Cue
In dynamic scenes, motion provides a "free" supervisory signal for decomposition. If a group of points moves together rigidly while the rest of the scene remains static, those points likely belong to the same object.
Kaizhi Yang and Hao Su demonstrate this in MovingParts. By combining spatial (Eulerian) and particle-based (Lagrangian) views of motion, their framework discovers rigid parts in a dynamic NeRF without any manual labels. This turns movement into a powerful cue for unsupervised segmentation, allowing the system to distinguish between, for example, a person's arm and their torso based solely on how they move.
Enforcing Physical and Geometric Coherence
Simply separating objects isn't enough; they must also respect physical boundaries. In a monolithic model, objects often "bleed" into each other or the floor.
To solve this, ObjectSDF++, developed by Qianyi Wu and Jianfei Cai, introduces an occlusion-aware opacity rendering scheme and a regularization term to prevent object collisions. Similarly, ObjectCarver by Gemmechu Hassena and Noah Snavely uses a "compactness loss" to eliminate floating artifacts and ensure that reconstructed objects have clean, usable boundaries.
Why Decomposition Matters
The transition from "blobs" to "objects" is what makes 3D reconstructions "simulation-ready." As argued in HoloScene, a digital twin is only useful if an agent can interact with its components. By decomposing scenes into discrete parts, we enable:
- Robotic Manipulation: Allowing a robot to identify the boundaries of a chair it needs to move (vMAP).
- Scene Editing: Moving, rotating, or removing objects via text prompts without warping the background (3DitScene).
- Reasoning: Enabling models to solve symbolic logic puzzles by identifying discrete entities (Slot Transformer).
By breaking the monolith, these researchers are turning static 3D pictures into interactive, actionable worlds.