Abstract
Panoptic Compositional Feature Field (PCFF) is a novel framework designed for high-quality, object-level editing of 3D scenes without the need for expensive, manually labeled data. By leveraging a metric learning approach, PCFF overcomes the "index inconsistency" problem inherent in automated 2D segmentation tools, where different IDs are assigned to the same object across multiple views. This enables robust scene decomposition and high-fidelity rendering from off-the-shelf network predictions.
Introduction: The Challenge of Editable NeRFs
Neural Radiance Fields (NeRFs) have transformed 3D scene rendering, yet they remain "holistic"—the entire scene is encoded within a single network's weights, complicating the manipulation of individual objects. To enable scene editability, researchers aim to decompose NeRFs into separate object components. However, most existing methods rely on ground-truth 3D instance annotations, which are costly to produce.
An alternative is using 2D segmentation networks (e.g., Mask2Former), but these suffer from 3D index inconsistency. As illustrated in Figure 1, an object may receive different IDs across different viewpoints, confusing standard training methods and resulting in poor object separation.
Methodology: The PCFF Framework
PCFF sidesteps the need for global ID consistency through three primary innovations integrated into its architecture, as shown in Figure 2.
1. Instance Quadruplet Metric Learning
The core contribution of PCFF is a metric learning strategy that avoids explicit ID matching. For any pixel (the "anchor"), the model learns to cluster features of the same object together while pushing features of different objects apart within each view. Over multiple views, the 3D feature space naturally organizes into distinct object clusters without requiring a unified global ID.
2. Semantic-Appearance Hierarchical Learning (SAHL)
To prevent semantic labeling from degrading visual quality, PCFF employs a hierarchical MLP structure. Shallower layers handle semantic logits, while deeper layers focus on complex appearance and density, ensuring that the two tasks do not interfere with one another.
3. Semantic-Guided Regional Refinement (SGRR)
PCFF utilizes "semantic entropy" to identify regions of high uncertainty, typically found at object boundaries. SGRR directs the model's attention to these regions, facilitating sharper edges and more precise rendering.
Experiments and Key Results
The framework was evaluated on the ScanNet, Replica, and ToyDesk datasets, demonstrating significant improvements over existing methods:
- Robustness to Inconsistent Labels: On ScanNet, PCFF achieved an IoU of 0.891, vastly outperforming SemanticNeRF (0.357) and ObjectSDF (0.486) when trained on inconsistent labels.
- Superior Rendering Quality: PCFF achieved a PSNR of 32.77 on the Replica dataset, surpassing SemanticNeRF's 31.74.
- Editing Versatility: The model supports various editing tasks, including object deletion, translation, duplication, and color modification, while maintaining multi-view consistency.
Discussion and Future Work
The primary insight of PCFF is prioritizing similarity over identity; by focusing on whether two points belong to the same object rather than identifying which object they are, the model achieves robust decomposition. Furthermore, decoupling semantic and appearance tasks through hierarchical depth prevents performance trade-offs.
While PCFF is robust to inconsistent labels, it remains dependent on the quality of the underlying 2D priors. Future work may involve integrating foundation models like Segment Anything (SAM) to enhance zero-shot editing capabilities and further reducing training time, which currently ranges from 9 to 15 hours.
Summary updated