Panoptic Compositional Feature Field for Editable Scene Rendering with Network-Inferred Labels via Metric Learning

Xinhua Cheng, Yanmin Wu, Mengxi Jia, Qian Wang, Jian Zhang

Abstract

Panoptic Compositional Feature Field (PCFF) is a novel framework designed for high-quality, object-level editing of 3D scenes without the need for expensive, manually labeled data. By leveraging a metric learning approach, PCFF overcomes the "index inconsistency" problem inherent in automated 2D segmentation tools, where different IDs are assigned to the same object across multiple views. This enables robust scene decomposition and high-fidelity rendering from off-the-shelf network predictions.

Introduction: The Challenge of Editable NeRFs

Neural Radiance Fields (NeRFs) have transformed 3D scene rendering, yet they remain "holistic"—the entire scene is encoded within a single network's weights, complicating the manipulation of individual objects. To enable scene editability, researchers aim to decompose NeRFs into separate object components. However, most existing methods rely on ground-truth 3D instance annotations, which are costly to produce.

An alternative is using 2D segmentation networks (e.g., Mask2Former), but these suffer from 3D index inconsistency. As illustrated in Figure 1, an object may receive different IDs across different viewpoints, confusing standard training methods and resulting in poor object separation.

Figure 1: The problem of 3D index inconsistency. Automated networks often assign different IDs to the same object across views, leading to incorrect object extraction compared to manually-labeled consistent annotations.

Methodology: The PCFF Framework

PCFF sidesteps the need for global ID consistency through three primary innovations integrated into its architecture, as shown in Figure 2.

Figure 2: The PCFF architecture overview, featuring hierarchical representation (SAHL), volume rendering, and multi-faceted supervision using semantic and instance labels with regional refinement (SGRR).

1. Instance Quadruplet Metric Learning

The core contribution of PCFF is a metric learning strategy that avoids explicit ID matching. For any pixel (the "anchor"), the model learns to cluster features of the same object together while pushing features of different objects apart within each view. Over multiple views, the 3D feature space naturally organizes into distinct object clusters without requiring a unified global ID.

2. Semantic-Appearance Hierarchical Learning (SAHL)

To prevent semantic labeling from degrading visual quality, PCFF employs a hierarchical MLP structure. Shallower layers handle semantic logits, while deeper layers focus on complex appearance and density, ensuring that the two tasks do not interfere with one another.

3. Semantic-Guided Regional Refinement (SGRR)

PCFF utilizes "semantic entropy" to identify regions of high uncertainty, typically found at object boundaries. SGRR directs the model's attention to these regions, facilitating sharper edges and more precise rendering.

Experiments and Key Results

The framework was evaluated on the ScanNet, Replica, and ToyDesk datasets, demonstrating significant improvements over existing methods:

  • Robustness to Inconsistent Labels: On ScanNet, PCFF achieved an IoU of 0.891, vastly outperforming SemanticNeRF (0.357) and ObjectSDF (0.486) when trained on inconsistent labels.
  • Superior Rendering Quality: PCFF achieved a PSNR of 32.77 on the Replica dataset, surpassing SemanticNeRF's 31.74.
  • Editing Versatility: The model supports various editing tasks, including object deletion, translation, duplication, and color modification, while maintaining multi-view consistency.

Discussion and Future Work

The primary insight of PCFF is prioritizing similarity over identity; by focusing on whether two points belong to the same object rather than identifying which object they are, the model achieves robust decomposition. Furthermore, decoupling semantic and appearance tasks through hierarchical depth prevents performance trade-offs.

While PCFF is robust to inconsistent labels, it remains dependent on the quality of the underlying 2D priors. Future work may involve integrating foundation models like Segment Anything (SAM) to enhance zero-shot editing capabilities and further reducing training time, which currently ranges from 9 to 15 hours.

Summary updated

Abstract

Despite neural implicit representations demonstrating impressive high-quality view synthesis capacity, decom-posing such representations into objects for instance-level editing is still challenging. Recent works learn object-compositional representations supervised by ground truth instance annotations and produce promising scene editing results. However, ground truth annotations are manually labeled and expensive in practice, which limits their usage in real-world scenes. In this work, we attempt to learn an object-compositional neural implicit representation for editable scene rendering by leveraging labels inferred from the off-the-shelf 2D panoptic segmentation networks instead of the ground truth annotations. We propose a novel framework named Panoptic Compositional Feature Field (PCFF), which introduces an instance quadruplet metric learning to build a discriminating panoptic feature space for reliable scene editing. In addition, we propose semantic-related strategies to further exploit the correlations between semantic and appearance attributes for achieving better rendering results. Experiments on multiple scene datasets including ScanNet, Replica, and ToyDesk demonstrate that our proposed method achieves superior performance for novel view synthesis and produces convincing real-world scene editing results.
BibTeX
@inproceedings{cheng2023panoptic,
  author = {Xinhua Cheng and Yanmin Wu and Mengxi Jia and Qian Wang and Jian Zhang},
  title = {{Panoptic Compositional Feature Field for Editable Scene Rendering with Network-Inferred Labels via Metric Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00479},
}
Figure