Introduction
While Neural Radiance Fields (NeRF) have revolutionized photorealistic 3D reconstruction, they typically treat a scene as a single, inseparable "cloud" of density and color. This makes it incredibly difficult to perform object-level manipulation—like moving a specific chair in a reconstructed room—without rebuilding the entire model.
DM-NeRF is a novel framework that reconstructs complex 3D scenes from 2D images while simultaneously decomposing them into individual, manipulatable objects. By adding a dedicated "object field" to the standard NeRF architecture, it allows users to translate, rotate, and scale 3D objects with high photorealism and robust handling of occlusions and collisions.
Methodology
The authors introduce two primary components integrated with a standard NeRF backbone: the Object Field and the Object Manipulator.
The Object Field
This is a parallel MLP branch that predicts a unique "object code" (a one-hot vector) for every 3D point in space. To train this without 3D ground truth, the system leverages 2D supervision through:
- Optimal Association: Since the number and order of objects vary across 2D views, the Hungarian algorithm matches 2D mask labels to 3D object predictions consistently.
- Empty Space Supervision: A critical innovation is supervising the "non-occupied" space. By estimating surface distances, the model learns to explicitly label empty areas, preventing "ghosting" artifacts when objects are moved.
Object Manipulator (Inverse Query Algorithm)
Instead of trying to "push" 3D geometry to a new location, the system uses an inverse query. When rendering a manipulated scene, the algorithm looks at a point in the new coordinate space and asks: "Where did this point come from in the original scene?" It then fetches the color and density from that original location. This approach naturally handles object collisions and visual occlusions.
Datasets and Key Insights
The researchers evaluated DM-NeRF on three distinct datasets: DM-SR (a new synthetic dataset for manipulation), Replica, and ScanNet.
A key insight is that 3D consistency acts as a denoising filter. The 3D object field is remarkably robust to noisy 2D labels because a 3D point must project consistently to multiple 2D views. In experiments with 80% corrupted training labels, DM-NeRF maintained high segmentation accuracy ($AP = 74.08$), while traditional 2D methods failed.
Furthermore, the model emphasizes the importance of "nothing". Most NeRF models focus on where matter is, but DM-NeRF demonstrates that to move objects cleanly, the model must have a strong grasp of where matter isn't. Their 3d_empty loss is essential for clean object boundaries.
Results and Limitations
On the ScanNet dataset, DM-NeRF achieved an Average Precision ($AP^{0.75}$) of 95.60, significantly outperforming 2D-based baselines like Mask-RCNN and Swin Transformer. Quantitative results for translation and rotation showed that DM-NeRF maintains much higher image quality (PSNR ~31-33) compared to Point-NeRF (PSNR ~23-25) when scenes are manipulated.
However, limitations remain:
- Lighting Inconsistency: Shadows and reflections do not update when objects are moved.
- Occlusion Artifacts: Moving objects may reveal "unseen" areas that the model hallucinations poorly.
- Static Scenes: The framework is currently designed for static environments.
DM-NeRF stands as one of the first methods to simultaneously reconstruct, decompose, manipulate, and render complex 3D scenes in a single pipeline.
Summary updated