Modern 3D reconstruction techniques like Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting have made photorealistic scene synthesis remarkably accessible. By optimizing continuous functions or dense particle clouds to match a set of 2D images, these methods capture intricate geometry and view-dependent lighting with stunning fidelity.
However, these representations suffer from a fundamental limitation: they are monolithic. To a standard NeRF, an entire room—including the walls, a table, and a coffee mug—is baked into a single, inseparable "soup" of density and color. The model has no concept of discrete entities. As noted by Karl Stelzner et al. in Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation, this lack of structural decomposition prevents autonomous systems from transitioning from passive scene observers to active agents that can manipulate, reason about, or interact with individual objects. If a robot wants to pick up the coffee mug, or a virtual reality user wants to rearrange the furniture, a monolithic representation offers no clean way to isolate where one object ends and another begins.
To bridge this gap, a major research direction has emerged focused on object-level scene decomposition. The goal is to untangle these continuous volumetric representations into discrete, semantically meaningful, and actionable 3D components.
Lifting 2D Semantics to 3D Feature Fields
The most intuitive way to introduce object awareness into a continuous 3D scene is to leverage what we already know from 2D computer vision. Foundation models like the Segment Anything Model (SAM), CLIP, or DINO are exceptionally good at identifying object boundaries and semantics in 2D images. The challenge lies in "lifting" this 2D understanding into 3D space.
To achieve this, researchers modify the standard volume rendering formulation. In a traditional NeRF, a ray is cast through the scene, and the network predicts a density $\sigma$ and an RGB color $c$ at every 3D coordinate. These values are integrated along the ray to render a pixel's color:
$$C(r) = \int w(t) c(t) dt$$
In a Distilled Feature Field (DFF), proposed by Sosuke Kobayashi et al. in Decomposing NeRF for Editing via Feature Field Distillation, the network is tasked with predicting an additional high-dimensional semantic feature vector $f$ at each 3D point. This feature is integrated along the ray in exactly the same manner as color:
$$F(r) = \int w(t) f(t) dt$$
By minimizing the difference between this rendered 3D feature map and the 2D feature map extracted by a pre-trained foundation model, the 3D field directly inherits the foundation model's semantic knowledge.
Once trained, users can query this 3D feature field using text or image prompts. For example, as demonstrated in Open-NeRF: Towards Open Vocabulary NeRF Decomposition by Hao Zhang et al., an open-vocabulary text query like "chair" can be mapped to the distilled embedding space, allowing the system to isolate the exact 3D region corresponding to that object. Similarly, Xinhang Liu et al. in Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-Aware Spatio-Temporal Sampling distill 2D masks into a 4D representation, enabling interactive, free-viewpoint object tracking across time from a single user click.
The Challenge of Multi-View Inconsistency
While lifting 2D features sounds straightforward, it introduces a major hurdle: multi-view inconsistency. 2D foundation models process each image independently. If you look at a chair from the front, a 2D segmentation network might assign it an ID of 4. If you view it from the back, the network might assign it an ID of 12, or fail to segment it entirely due to lighting changes or occlusions.
If you naively attempt to distill these conflicting 2D labels into a single 3D field, the optimization process fails to converge, resulting in blurry, ambiguous semantic boundaries.
To solve this "index inconsistency" problem, researchers have introduced several clever alignment strategies:
- Metric Learning: Instead of forcing the 3D network to predict arbitrary, view-dependent 2D IDs, Xinhua Cheng et al. in Panoptic Compositional Feature Field for Editable Scene Rendering with Network-Inferred Labels via Metric Learning employ a contrastive metric learning loss. This forces the network to map points belonging to the same physical object close together in a continuous embedding space, regardless of the noisy labels provided by off-the-shelf 2D tools.
- View-Consistency Regularization: In Enforcing View-Consistency in Class-Agnostic 3D Segmentation Fields, Corentin Dumery et al. introduce specialized matching losses that explicitly align 2D mask proposals across different viewpoints before they are baked into the 3D segmentation field.
- Geometric Propagation: Rather than relying purely on semantics, Xinhang Liu et al. in Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation use the underlying geometric consistency of the radiance field itself to propagate sparse labels across views, penalizing incorrect object assignments with a bidirectional loss.
Compositional Fields: Hard Partitioning by Design
Distilling feature fields adds a semantic "layer" on top of a monolithic network, but the underlying geometry remains unified. An alternative, more radical approach is to abandon the single-network architecture entirely. Instead, the scene can be modeled as a composition of independent neural fields or primitives.
Under this paradigm, every object in the scene is granted its own dedicated, localized model. For instance, in vMAP: Vectorised Object Mapping for Neural Field SLAM, Shikun Liu et al. represent a scene using dozens of tiny, independent MLPs—one for each object. By leveraging vectorized training, these models are optimized in parallel on a single GPU. Because each object is bounded by its own independent field, decomposition is guaranteed by design, allowing a robot to instantly isolate a specific chair or table.
Similarly, single-author work by Hao Su, Nerflets: Local Radiance Fields for Efficient Structure-Aware 3D Scene Representation from 2D Supervision, replaces the global network with a collection of local radiance field primitives. This structural decomposition not only enables clean object-level editing but also drastically improves computational efficiency.
For explicit point-based representations like 3D Gaussian Splatting, which inherently lack structural grouping, post-hoc partitioning can be achieved geometrically. Umangi Jain et al. demonstrate in GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splatting that by treating the 3D Gaussians as nodes in a graph, classic graph-cut algorithms can cleanly separate foreground objects from the background based on sparse user interactions. Taking structural abstraction even further, Tom Monnier et al. introduce Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives, which bypasses the typical "soup of particles" entirely, decomposing scenes directly into a compact set of interpretable, textured geometric blocks that are instantly ready for physics engines.
Resolving Scale Ambiguity and Occlusions
As the field has matured, researchers have begun tackling the deeper physical and semantic ambiguities inherent to object decomposition.
The Ambiguity of Scale
What exactly constitutes an "object"? If a coffee mug sits on a table, is the mug a separate entity, or is it part of the table? What about the mug's handle?
To resolve this philosophical and practical challenge, Chung Min Kim et al. developed GARField: Group Anything with Radiance Fields. Instead of forcing a hard, single-level segmentation, the authors optimize a scale-conditioned affinity field. This allows a single 3D point to belong to multiple nested clusters simultaneously. By specifying a physical scale of interest, a user or robot can dynamically choose whether to interact with a tiny sub-part, a discrete object, or an entire group of items.
The "Hole" Problem (Occlusion)
When you physically remove a chair from a room, you reveal the floor and wall that were previously hidden behind it. In a standard multi-view capture, these regions are completely unobserved. If you simply "peel" an object out of a standard NeRF, you are left with an empty, hollow shell or severe visual artifacts in the background.
To achieve true, simulation-ready decomposition, models must be able to "hallucinate" what lies behind occlusions. In Learning Unified Decompositional and Compositional NeRF for Editable Novel View Synthesis, Yuxin WANG et al. tackle this by integrating 2D in-painting networks into a two-stage rendering pipeline to fill in occluded backgrounds.
More recently, researchers have turned to powerful generative priors. Junfeng Ni et al. introduce Decompositional Neural Scene Reconstruction with Generative Diffusion Prior, which leverages pre-trained 2D diffusion models via a visibility-guided Score Distillation Sampling (SDS) mechanism. When an object is isolated, the diffusion prior acts as a geometric and textural "imagination," completing the hidden, unobserved back-sides of objects and the backgrounds behind them, ensuring they remain watertight and realistic from any angle.
Conclusion
By moving away from monolithic volumes and toward object-compositional architectures, this research direction fundamentally changes what neural fields can do. Single-author work by Jiakai Cao, Structured-NeRF: Hierarchical Scene Graph with Neural Representation, showcases the ultimate trajectory of this field: organizing neural scenes into hierarchical scene graphs that understand both semantic and physical relationships.
Whether achieved by distilling view-consistent 2D features, training parallel networks for discrete assets, or leveraging generative diffusion models to complete hidden geometry, object-level decomposition transforms neural reconstructions from static, passive pictures into dynamic, interactive, and simulation-ready digital worlds.
Key Foundational Work
Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation (2021) — contributed 1 concept elements to this direction.
Neural Volumetric Object Selection (2022) — contributed 1 concept elements to this direction.
Decomposing NeRF for Editing via Feature Field Distillation (2022) — contributed 1 concept elements to this direction.