OV-NeRF: Open-Vocabulary Neural Radiance Fields With Vision and Language Foundation Models for 3D Semantic Understanding

Qing Li (李庆), Guibiao Liao, Kaichen Zhou, Zhenyu Bao, Kanglin Liu, Qing Li

Introduction

Neural Radiance Fields (NeRFs) have revolutionized 3D scene reconstruction, providing photorealistic renderings from sparse 2D images. However, standard NeRFs lack semantic awareness—they do not inherently "know" what objects are present in a scene. OV-NeRF addresses this gap by integrating Vision-Language Models (VLMs) to enable open-vocabulary 3D semantic understanding without requiring manual labels.

The Challenge: Noise and Inconsistency

Current approaches often rely on models like CLIP to provide semantic labels. However, CLIP faces two significant hurdles in 3D contexts:

  1. Coarseness: Designed for image-level classification, CLIP produces noisy, blurry "relevancy maps" that fail to capture precise object boundaries.
  2. View Inconsistency: 2D labels from CLIP often vary across different viewpoints, leading to "foggy" or inaccurate 3D semantic fields when integrated into a NeRF.

Methodology

OV-NeRF introduces two core components to overcome these limitations, built upon the efficient TensoRF architecture:

Region Semantic Ranking (RSR)

To resolve the issue of blurry boundaries, the framework utilizes the Segment Anything Model (SAM). While SAM provides high-quality geometric boundaries, it lacks semantic labels. RSR bridges this gap by using SAM’s precise region proposals to aggregate and "vote" on CLIP labels. This ensures that the semantic field respects sharp object edges.

Cross-view Self-enhancement (CSE)

To address view inconsistency, OV-NeRF leverages the inherent 3D consistency of the radiance field:

  • Iterative Pseudo Map Update: After initial training, the model renders its own semantic maps from the 3D volume. These renders are naturally more consistent than raw 2D CLIP outputs and are used as "pseudo-labels" for further refinement.
  • Novel Pseudo Map Synthesis: By generating labels for unseen viewpoints, the model provides additional spatial cues to the training process, effectively using the NeRF as its own teacher.
Comparison of relevancy maps and final predictions between vanilla CLIP, 3DOVS, and OV-NeRF.

Experimental Results

The authors evaluated OV-NeRF on the Replica, ScanNet, and Mip-NeRF360 datasets. The results demonstrate a significant leap in performance:

  • Accuracy Gains: OV-NeRF achieved a 20.31% improvement in mean Intersection-over-Union (mIoU) on the Replica dataset and an 18.42% improvement on ScanNet compared to the previous state-of-the-art method, 3DOVS.
  • Visual Quality: Qualitative results show much sharper object boundaries and a marked reduction in semantic "bleeding" between adjacent objects, such as tables and floors.
  • Flexibility: The system maintains open-vocabulary capabilities, allowing users to query the 3D scene for arbitrary text descriptions like "vintage lamp" or "wooden stool."

Conclusion

OV-NeRF demonstrates that the synergy between foundation models like CLIP and SAM, combined with the self-correcting nature of 3D radiance fields, can drastically improve semantic scene understanding. By filtering 2D noise through 3D consensus, it paves the way for more robust autonomous navigation and interactive 3D environments.

Summary updated

BibTeX
@article{2024ov,
  author = {Qing Li (李庆) and Guibiao Liao and Kaichen Zhou and Zhenyu Bao and Kanglin Liu and Qing Li},
  title = {{OV-NeRF: Open-Vocabulary Neural Radiance Fields With Vision and Language Foundation Models for 3D Semantic Understanding}},
  journal = {IEEE TCSVT},
  year = {2024},
  doi = {10.1109/tcsvt.2024.3439737},
}
Figure