Cues3d: Unleashing the Power of Sole Nerf for Consistent and Unique Instances in Open-Vocabulary 3d Panoptic Segmentation

Xue Feng, Wenzhuang Xu, Guofeng Zhong, Anlong Ming, Nicu SebeSSRN 2024 · DOI

TL;DR

This paper introduces Cues3D, a novel approach for open-vocabulary 3D panoptic segmentation that leverages the inherent geometric consistency of Neural Radiance Fields (NeRFs) to achieve accurate and unique instance identification without relying on external 3D data or complex pre-association steps. By introducing a three-stage training framework, Cues3D effectively refines instance masks and their identities, outperforming existing methods on challenging benchmarks.

What Problem Does This Paper Address?

Open-vocabulary 3D panoptic segmentation, the task of identifying and categorizing all objects and background elements in a 3D scene using arbitrary text descriptions, is a crucial component for embodied AI and robotics. While current state-of-the-art methods often combine 2D image segmentation with 3D geometric information, they face challenges with high-fidelity 3D data and maintaining consistent instance identities across different views. Methods relying solely on 2D images struggle with view-inconsistency, leading to multiple IDs for the same object. This paper addresses the challenge of achieving globally consistent and unique 3D instance segmentation using only 2D images, by exploiting the implicit scene representation learned by NeRFs.

Methods

Cues3D employs a three-stage training framework built upon NeRF:

  1. Initialization: The NeRF model is trained with view-inconsistent 2D instance labels. This phase aims to encode the scene's implicit 3D geometry and achieve a rough, but globally inconsistent, instance field. A Hungarian training scheme is used to align predicted instance IDs with ground truth labels.
  2. Disambiguation: This crucial stage leverages the 3D geometry learned in the first phase to resolve instance ID inconsistencies. It involves extracting 3D masks from rendered NeRF outputs, performing voxel-based nearest neighbor matching between these masks, and then unifying overlapping instances to assign unique global IDs. This process is visualized in Figure 5 and detailed in Algorithm 1.
  3. Refinement: The corrected and disambiguated instance IDs are used to re-train the NeRF model. This phase ensures a stable and consistent instance field across all views, leading to highly accurate and globally unique 3D instance segmentation.

Datasets

The proposed Cues3D method is evaluated on several widely used 3D scene understanding datasets:

  • ScanNet v2: A popular dataset with 3D point clouds, 2D images, and detailed instance/semantic annotations.
  • ScanNet200: An extension of ScanNet with more classes and annotations.
  • Replica: A dataset known for its high-quality indoor scene reconstructions, often used in Novel View Synthesis (NVS) research.
  • ScanNet++: A recently proposed dataset with improved image quality and annotation accuracy compared to ScanNet v2.

Key Insights and Ideas

  • NeRF as a Geometry Engine for Instance Consistency: The paper's core insight is that NeRF's implicit 3D representation inherently captures global scene geometry, which can be leveraged to enforce cross-view consistency for 3D instances without explicit pre-association or contrastive losses.
  • Disambiguation via 3D Geometry: Differentiating objects with the same ID from local 2D views is difficult, but Cues3D shows that using the 3D geometry learned by NeRF makes this disambiguation much more tractable.
  • Synergy of Rendered and Ground Truth Masks: The disambiguation phase effectively combines the strengths of NeRF-rendered instance masks (consistent IDs but potentially inaccurate boundaries) and ground truth 2D instance masks (accurate boundaries but inconsistent IDs) to produce accurate and globally consistent 3D instance masks.
  • Zero-Shot Open-Vocabulary Capability: Cues3D's ability to perform open-vocabulary segmentation by leveraging VLMs like CLIP for semantic projection on top of its consistent instance segmentation is a significant advantage.

Key Results

Cues3D demonstrates strong performance across various 3D tasks:

  • 3D Instance Segmentation: On ScanNet++, Cues3D achieves an 11.1% higher AP than SAI3D (a 2D-3D method) using only 2D images. When augmented with 3D superpoint, it surpasses other 2D-3D methods, achieving a 4.9% improvement over MaskClustering on ScanNet v2 and 6.9% on ScanNet200.
  • 3D Panoptic Segmentation: Cues3D achieves competitive results, outperforming PVLFF by +17.3% PQscene on ScanNet v2 and +1.7% on Replica, even when using an earlier version of CLIP. With FC-CLIP, it further improves PQscene by 6.7% and 3.5% on ScanNet v2 and Replica, respectively.
  • 3D Semantic Segmentation: Cues3D achieves a 2.3% higher mIoU than OpenScene on ScanNet v2 and outperforms OpenScene by 11.7% mIoU and 14.2% mAcc when using FC-CLIP.
  • Qualitative Results: Visualizations show Cues3D producing highly consistent and integrated instances, outperforming other methods in instance integrity and semantic segmentation accuracy.

Limitations and Future Work

The authors acknowledge two main limitations:

  • Reliance on NeRF Reconstruction Quality: The disambiguation process depends on NeRF-generated 3D representations. Poor reconstruction quality (e.g., from low-quality sensors or fast-motion scenes) could disrupt this process, limiting applicability in challenging scenarios.
  • Lack of Hierarchical Instance Segmentation: The current approach does not support hierarchical 3D instance segmentation, which could limit flexible 3D semantic predictions and applications in complex robot-environment interactions.

Future work directions include:

  • Improving NVS Methods: Further enhancing the performance of Cues3D by integrating with emerging, higher-performing NVS methods.
  • Online 3D Scene Understanding: Developing an online system that continuously updates 3D instance representations of scenes, enabling local understanding and efficient updates for agents.
  • Multi-modal Integration: Incorporating data from other sensors like LiDAR to improve robustness in diverse lighting and motion conditions.
  • Real-time Deployment: Optimizing efficiency for deployment on edge devices in resource-constrained environments.

Summary updated

BibTeX
@misc{feng2024cues,
  author = {Xue Feng and Wenzhuang Xu and Guofeng Zhong and Anlong Ming and Nicu Sebe},
  title = {{Cues3d: Unleashing the Power of Sole Nerf for Consistent and Unique Instances in Open-Vocabulary 3d Panoptic Segmentation}},
  year = {2024},
  doi = {10.2139/ssrn.4966991},
  howpublished = {SSRN},
}
Figure