DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images

Bing Wang, Lu Chen, Bo Yang

Introduction

While Neural Radiance Fields (NeRF) have revolutionized photorealistic 3D reconstruction, they typically treat a scene as a single, inseparable "cloud" of density and color. This makes it incredibly difficult to perform object-level manipulation—like moving a specific chair in a reconstructed room—without rebuilding the entire model.

DM-NeRF is a novel framework that reconstructs complex 3D scenes from 2D images while simultaneously decomposing them into individual, manipulatable objects. By adding a dedicated "object field" to the standard NeRF architecture, it allows users to translate, rotate, and scale 3D objects with high photorealism and robust handling of occlusions and collisions.

Overview of the DM-NeRF pipeline, showing the transition from 2D views to a manipulatable 3D scene representation.

Methodology

The authors introduce two primary components integrated with a standard NeRF backbone: the Object Field and the Object Manipulator.

The Object Field

This is a parallel MLP branch that predicts a unique "object code" (a one-hot vector) for every 3D point in space. To train this without 3D ground truth, the system leverages 2D supervision through:

  • Optimal Association: Since the number and order of objects vary across 2D views, the Hungarian algorithm matches 2D mask labels to 3D object predictions consistently.
  • Empty Space Supervision: A critical innovation is supervising the "non-occupied" space. By estimating surface distances, the model learns to explicitly label empty areas, preventing "ghosting" artifacts when objects are moved.
Detailed architecture of the Object Field Component integrated with NeRF, showing the generation of density, color, and object codes.

Object Manipulator (Inverse Query Algorithm)

Instead of trying to "push" 3D geometry to a new location, the system uses an inverse query. When rendering a manipulated scene, the algorithm looks at a point in the new coordinate space and asks: "Where did this point come from in the original scene?" It then fetches the color and density from that original location. This approach naturally handles object collisions and visual occlusions.

Datasets and Key Insights

The researchers evaluated DM-NeRF on three distinct datasets: DM-SR (a new synthetic dataset for manipulation), Replica, and ScanNet.

A key insight is that 3D consistency acts as a denoising filter. The 3D object field is remarkably robust to noisy 2D labels because a 3D point must project consistently to multiple 2D views. In experiments with 80% corrupted training labels, DM-NeRF maintained high segmentation accuracy ($AP = 74.08$), while traditional 2D methods failed.

Furthermore, the model emphasizes the importance of "nothing". Most NeRF models focus on where matter is, but DM-NeRF demonstrates that to move objects cleanly, the model must have a strong grasp of where matter isn't. Their 3d_empty loss is essential for clean object boundaries.

Results and Limitations

On the ScanNet dataset, DM-NeRF achieved an Average Precision ($AP^{0.75}$) of 95.60, significantly outperforming 2D-based baselines like Mask-RCNN and Swin Transformer. Quantitative results for translation and rotation showed that DM-NeRF maintains much higher image quality (PSNR ~31-33) compared to Point-NeRF (PSNR ~23-25) when scenes are manipulated.

However, limitations remain:

  • Lighting Inconsistency: Shadows and reflections do not update when objects are moved.
  • Occlusion Artifacts: Moving objects may reveal "unseen" areas that the model hallucinations poorly.
  • Static Scenes: The framework is currently designed for static environments.

DM-NeRF stands as one of the first methods to simultaneously reconstruct, decompose, manipulate, and render complex 3D scenes in a single pipeline.

Summary updated

Abstract

In this paper, we study the problem of 3D scene geometry decomposition and manipulation from 2D views. By leveraging the recent implicit neural representation techniques, particularly the appealing neural radiance fields, we introduce an object field component to learn unique codes for all individual objects in 3D space only from 2D supervision. The key to this component is a series of carefully designed loss functions to enable every 3D point, especially in non-occupied space, to be effectively optimized even without 3D labels. In addition, we introduce an inverse query algorithm to freely manipulate any specified 3D object shape in the learned scene representation. Notably, our manipulation algorithm can explicitly tackle key issues such as object collisions and visual occlusions. Our method, called DM-NeRF, is among the first to simultaneously reconstruct, decompose, manipulate and render complex 3D scenes in a single pipeline. Extensive experiments on three datasets clearly show that our method can accurately decompose all 3D objects from 2D views, allowing any interested object to be freely manipulated in 3D space such as translation, rotation, size adjustment, and deformation.
BibTeX
@misc{wang2022dm,
  author = {Bing WANG and Lu Chen and Bo Yang},
  title = {{DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images}},
  year = {2022},
  eprint = {2208.07227},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.07227},
}
Figure