DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images

Bing Wang, Lu Chen, Bo Yang

Abstract

Neural Radiance Fields (NeRF) have set new standards for photorealistic 3D reconstruction, yet they often lack the ability to decompose scenes into individual, editable components. We present DM-NeRF, a novel framework that achieves 3D scene geometry decomposition and manipulation using only 2D supervision. By integrating an "object field" component that learns unique codes for individual objects and an "inverse query" algorithm for spatial manipulation, DM-NeRF allows for precise translation, rotation, and scaling of objects within a complex scene. Our approach effectively handles spatial collisions and visual occlusions, outperforming existing methods in both decomposition accuracy and rendering quality.

Introduction

The primary limitation of standard NeRF models is their holistic representation of scenes as continuous volumes of density and color. While this enables stunning view synthesis, it prevents the granular editing required for interactive applications. To move a chair or resize a table, the model must first understand the boundaries of these objects in 3D space. DM-NeRF addresses this by learning a view-independent 3D identity for every point, bridging the gap between static reconstruction and dynamic scene manipulation.

The DM-NeRF Framework

The core of DM-NeRF is a dual-branch architecture that extends the standard NeRF pipeline. While the primary branch predicts volume density and RGB color, a parallel "Object Field" branch predicts a unique object code for every 3D point.

The DM-NeRF pipeline: from 2D views to a decomposed scene representation and object manipulation.

This architecture allows the model to reconstruct the scene's geometry while simultaneously assigning semantic identities to its components. The training process relies solely on 2D images and instance masks. By projecting the 3D object codes back into 2D views, the model can be optimized using standard segmentation losses.

Key Innovations: Object Fields and Empty Space

A significant challenge in 3D decomposition from 2D views is the ambiguity of "empty" space. Without explicit 3D labels, points in non-occupied space might incorrectly inherit object identities, leading to "ghosting" artifacts. DM-NeRF introduces an additional dimension in the object code specifically for empty space. By supervising this dimension using estimated surface distances, the model learns to distinguish between solid objects and the air between them.

Hungarian matching is used to maintain consistency between different 2D views during the training of the object field.

To ensure that object identities remain consistent across different camera angles, the model employs a matching strategy (such as Hungarian matching) to align predicted masks with ground truth labels across multiple views. This multi-view consistency is what allows DM-NeRF to be robust against noisy 2D labels; the 3D representation must satisfy all views simultaneously, effectively filtering out random errors in individual 2D masks.

Object Manipulation via Inverse Query

Once the scene is decomposed, DM-NeRF enables manipulation through an "inverse query" algorithm. Rather than physically moving the 3D points—which would require re-training or complex mesh deformation—the system transforms the query coordinates. When rendering a scene where an object has been moved, the model takes a point in the new target location and calculates its original position in the source scene. It then retrieves the corresponding color and density. This method naturally handles occlusions and allows for the detection of collisions by checking if a moved object's volume overlaps with another occupied region.

Experimental Evaluation

The researchers tested DM-NeRF across three datasets: the new synthetic DM-SR dataset, the high-fidelity Replica dataset, and the real-world ScanNet dataset.

  • Decomposition Accuracy: DM-NeRF achieved near-perfect results on synthetic data (99.80 [email protected]), significantly outperforming 2D-only baselines like Mask-RCNN.
  • Robustness: The model maintained high performance even when 80% of the training masks were intentionally corrupted, demonstrating the power of 3D-consistent learning.
  • Manipulation Quality: Compared to point-based methods like Point-NeRF, DM-NeRF produced sharper images without the "holes" or artifacts common in explicit geometric manipulations.

Conclusion

DM-NeRF represents a significant step toward interactive neural scene representations. By successfully decoupling object identity from spatial geometry using only 2D supervision, it enables a wide range of editing capabilities—from simple translations to complex deformations—while maintaining the photorealistic quality of NeRF. Future work will focus on disentangling lighting from geometry to ensure that shadows and reflections update realistically as objects are moved.

Summary updated

Abstract

In this paper, we study the problem of 3D scene geometry decomposition and manipulation from 2D views. By leveraging the recent implicit neural representation techniques, particularly the appealing neural radiance fields, we introduce an object field component to learn unique codes for all individual objects in 3D space only from 2D supervision. The key to this component is a series of carefully designed loss functions to enable every 3D point, especially in non-occupied space, to be effectively optimized even without 3D labels. In addition, we introduce an inverse query algorithm to freely manipulate any specified 3D object shape in the learned scene representation. Notably, our manipulation algorithm can explicitly tackle key issues such as object collisions and visual occlusions. Our method, called DM-NeRF, is among the first to simultaneously reconstruct, decompose, manipulate and render complex 3D scenes in a single pipeline. Extensive experiments on three datasets clearly show that our method can accurately decompose all 3D objects from 2D views, allowing any interested object to be freely manipulated in 3D space such as translation, rotation, size adjustment, and deformation.
BibTeX
@misc{wang2022dm,
  author = {Bing Wang and Lu Chen and Bo Yang},
  title = {{DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images}},
  year = {2022},
  eprint = {2208.07227},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.07227},
}
Figure