Standard Neural Radiance Fields (NeRFs) and 3D Gaussian Splatting represent scenes as monolithic entities. They map global 3D coordinates to color and density or optimize a single, interconnected cloud of primitives. While this achieves stunning photorealism, it creates a fundamental limitation: the scene is a continuous "black box." If you want to move a single chair, change the trajectory of a vehicle, or swap an object, you cannot easily do so because the entire environment's geometry and appearance are baked together.
To overcome this, a powerful paradigm has emerged: compositional neural fields. Instead of modeling a scene with a single global network, the environment is factorized into a collection of independent, modular neural assets.
The Core Intuition: Decoupling Coordinate Frames
To make a neural scene editable, we must give each object its own local coordinate system and its own dedicated neural representation.
Consider a dynamic scene containing a moving object and a static background. In a traditional neural field, a network must learn a complex, time-varying mapping to capture the object's deformation and motion through space. In contrast, frameworks like STaR (Yuan et al.) and NeuMan (Jiang et al.) factorize the scene into two separate fields: one for the static background environment and another for the moving subject.
Monolithic Representation:
[ Global Coordinates ] ---> [ Single Large Network ] ---> [ Color / Density ]
Compositional Representation:
---> [ Local Frame A ] ---> [ Object Network A ] ---|
[ Global Coordinates ] |---> [ Composition Layer ]
---> [ Background Frame ] -> [ Background Network ] -|
Because the object is stored in its own local coordinate space, its motion through the world can be represented simply as a rigid 3D transformation matrix (rotation and translation) applied to its coordinate frame over time. This separation enables counterfactual animation. You can change the object's trajectory to a path never observed in the original source video, and the system can still render it accurately because the object's internal appearance representation remains invariant to its global position.
Recomposing the Scene via Volume Rendering
Once a scene is split into multiple independent neural fields, how do we reassemble them into a single, coherent image? We cannot evaluate them in isolation; we must compose them along each camera ray.
In standard volume rendering, a ray marches through space, sampling density and color at discrete points. In a compositional framework, such as GIRAFFE introduced by Geiger or the architecture proposed by Zhong et al., the ray queries multiple networks simultaneously. For any given point in space, the system transforms the global ray coordinates into the local coordinate frames of nearby object bounding boxes.
To combine these outputs without creating visual artifacts like ghosting or blending seams, the densities are aggregated probabilistically. As demonstrated by Stelzner et al., the total scene density at any point along a ray can be treated as the sum of individual object densities:
$$\sigma_{\text{total}}(\mathbf{x}) = \sum_{i} \sigma_i(\mathbf{x}_i)$$
The samples are then depth-sorted along the ray to compute the final pixel color. This explicit composition ensures that foreground objects correctly occlude the background, allowing practitioners to cleanly insert, duplicate, or relocate elements without retraining the underlying models.
Unsupervised Object Discovery
A major challenge is training these models without requiring tedious manual 3D annotations or bounding boxes. Researchers have developed several clever inductive biases to discover object boundaries automatically from raw data:
- Translation Invariance: Yu et al. leverage the principle that an object's identity and shape should remain constant regardless of where it is placed in the scene. By explicitly separating "what" an object looks like from "where" it is located, the model learns to isolate discrete entities without human labels.
- Motion Cues: Objects that move independently from the background provide a strong signal for decomposition. Chen et al. demonstrate how lifting 2D motion-based segmentations into 3D neural fields allows the system to partition scenes cleanly based purely on object dynamics.
- Distilling 2D Foundation Models: Instead of relying solely on geometric cues, modern frameworks distill semantic knowledge from 2D vision models (like SAM or CLIP). For instance, Dumery et al. show how noisy, view-inconsistent 2D masks can be regularized into clean, view-consistent 3D object slots.
Why Modularity Changes Everything
By moving from monolithic to compositional representations, neural fields gain several transformative capabilities that make them highly practical for downstream applications:
Clean, Localized Editing
In a modular field, editing an object is a local operation. As Bartrum et al. show with ReplaceAnything3D, you can erase and replace a specific foreground object using text prompts without causing texture leakage or degrading the surrounding background.
Cross-Scene Asset Transfer
Because objects are stored as self-contained neural assets, they can be extracted and placed into entirely new environments. Xiangli et al. demonstrate this by mining assets from one ground feature plane and reconfiguring them in another, enabling the construction of vast, customizable 3D asset libraries.
High-Fidelity Simulation for Robotics and Autonomy
For applications like autonomous driving and robotic manipulation, compositional fields act as "differentiable sandboxes." Simulators can insert interactive traffic participants into real-world captured backgrounds (Song), generate counterfactual LiDAR returns without ghosting artifacts (as explored in Dynamic LiDAR Re-simulation using Compositional Neural Fields), or allow robots to imagine novel manipulation scenarios for imitation learning (Barcellona et al.).
Key Foundational Work
STaR: Self-Supervised Tracking and Reconstruction of Rigid Objects in Motion With Neural Rendering (2021) — contributed 2 concept elements to this direction.