Open-NeRF: Towards Open Vocabulary NeRF Decomposition

Hao Zhang, Fang Li, Narendra Ahuja

Abstract

In this paper, we address the challenge of decomposing Neural Radiance Fields (NeRF) into objects from an open vocabulary, a critical task for object manipulation in 3D reconstruction and view synthesis. Current techniques for NeRF decomposition involve a trade-off between the flexibility of processing open-vocabulary queries and the accuracy of 3D segmentation. We present Open-vocabulary Embedded Neural Radiance Fields (Open-NeRF), which leverage large-scale, off-the-shelf segmentation models like the Segment Anything Model (SAM) and introduce an integrate-and-distill paradigm with hierarchical embeddings to achieve both the flexibility of open-vocabulary querying and 3D segmentation accuracy. Open-NeRF first utilizes large-scale foundation models to generate hierarchical 2D mask proposals from varying viewpoints. These proposals are then aligned via tracking approaches, integrated within the 3D space, and subsequently distilled into the 3D field. This process ensures consistent recognition and granularity of objects from different viewpoints, even in challenging scenarios involving occlusion and indistinct features. Our experimental results show that the proposed Open-NeRF outperforms state-of-the-art methods such as LERF and FFD in open-vocabulary scenarios.

Introduction

Neural Radiance Fields (NeRFs) have revolutionized photorealistic 3D reconstruction. However, they typically represent scenes as continuous fields of density and color, lacking an inherent understanding of individual objects. For applications like robotics, virtual reality, and 3D scene editing, it is essential to decompose these fields into distinct, manipulatable entities.

Overview of the Open-NeRF architecture and its integrate-and-distill pipeline.

Existing decomposition methods face a significant "flexibility vs. accuracy" trade-off. Closed-vocabulary models are restricted to predefined categories, while open-vocabulary models using CLIP often struggle with imprecise 3D boundaries. Furthermore, directly applying 2D segmentation models like SAM leads to viewpoint inconsistency—an object might be segmented as a whole from one angle but as multiple parts from another. Open-NeRF addresses these issues by ensuring 3D consistency through a novel integration process before distillation.

Methodology: The Integrate-and-Distill Paradigm

The core of Open-NeRF is the "integrate-and-distill" paradigm, which prioritizes 3D consistency by processing 2D data before it is incorporated into the neural field.

Comparison between current feature extraction methods and the proposed integrate-and-distill approach.

1. 2D Knowledge Extraction and Alignment

The system first generates 2D mask proposals across multiple viewpoints using GroundingDINO and SAM. To overcome the "granularity noise" inherent in 2D models, a tracking model (SAM-Track) is employed to link masks of the same object across different frames, ensuring that a "Lego excavator" is recognized as the same entity regardless of the camera angle.

2. Knowledge Integration and Distillation

Once aligned, CLIP embeddings for all 2D masks belonging to a specific 3D object are fused to create a stable "consensus" embedding. This integrated knowledge is then distilled into an auxiliary open-vocabulary embedding field within the NeRF. Crucially, this embedding field is optimized independently of the core geometry, preventing semantic learning from degrading visual reconstruction quality.

3. Hierarchical Embedding

To support queries at various scales (e.g., "the car" vs. "the wheel"), Open-NeRF learns hierarchical embeddings at three levels: object, part, and background. For background regions, the system leverages LSeg to ensure comprehensive scene coverage.

Experimental Evaluation

Open-NeRF was evaluated on the Mip-NeRF 360 dataset (Kitchen and Garden scenes) and several custom real-world scenes (Toy, Car, Desktop) featuring long-tail objects and complex occlusions.

Quantitative Results

Open-NeRF demonstrates substantial improvements over state-of-the-art methods like LERF and FFD. In the 'Toy-1' scene, Open-NeRF achieved a mean Area Under Precision-Recall Curve (AUPRC) of 92.8%, representing a 26.3% improvement over LERF (66.5%). In the 'Car' scene, it reached 89.7% AUPRC, outperforming LERF by 32.2%.

Qualitative Performance

Qualitatively, Open-NeRF produces sharp, distinct masks for complex objects. Unlike previous methods that often produce blurry heatmaps bleeding into the environment, Open-NeRF maintains clear boundaries. This high-quality decomposition enables precise query-guided manipulations, such as changing the texture of a specific object based on a text prompt.

Conclusion

Open-NeRF provides a robust solution for open-vocabulary NeRF decomposition. By shifting from direct 2D-to-3D distillation to an integrate-and-distill approach, the model achieves superior consistency and accuracy. While the current pipeline involves significant computational overhead due to the reliance on multiple foundation models, it sets a new benchmark for 3D scene understanding and interaction in open-world environments.

Summary updated

Abstract

In this paper, we address the challenge of decomposing Neural Radiance Fields (NeRF) into objects from an open vocabulary, a critical task for object manipulation in 3D reconstruction and view synthesis. Current techniques for NeRF decomposition involve a trade-off between the flexibility of processing open-vocabulary queries and the accuracy of 3D segmentation. We present, Open-vocabulary Embedded Neural Radiance Fields (Open-NeRF), that leverage large-scale, off-the-shelf, segmentation models like the Segment Anything Model (SAM) and introduce an integrate-and-distill paradigm with hierarchical embeddings to achieve both the flexibility of open-vocabulary querying and 3D segmentation accuracy. Open-NeRF first utilizes large-scale foundation models to generate hierarchical 2D mask proposals from varying viewpoints. These proposals are then aligned via tracking approaches and integrated within the 3D space and subsequently distilled into the 3D field. This process ensures consistent recognition and granularity of objects from different viewpoints, even in challenging scenarios involving occlusion and indistinct features. Our experimental results show that the proposed Open-NeRF 1 outperforms state-of-the-art methods such as LERF [16] and FFD [18] in open-vocabulary scenarios. Open-NeRF offers a promising solution to NeRF decomposition, guided by open-vocabulary queries, enabling novel applications in robotics and vision-language interaction in open-world 3D scenes. Please find the code at https://github.com/haoz19/Open-NeRF.
BibTeX
@inproceedings{zhang2024open,
  author = {Hao Zhang and Fang Li and Narendra Ahuja},
  title = {{Open-NeRF: Towards Open Vocabulary NeRF Decomposition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=0xI0U3nDWX},
}
Figure