Introduction
Neural Radiance Fields (NeRF) have revolutionized 3D scene reconstruction, yet they remain difficult to edit due to their implicit nature. Conversely, 2D Generative Adversarial Networks (GANs), such as StyleGAN, offer intuitive latent spaces for editing but lack 3D consistency across varying viewpoints. NeRFEditor bridges this gap by introducing a framework that combines the high-fidelity editability of 2D GANs with the structural consistency of NeRF, enabling 360-degree scene manipulation.
Methodology
The NeRFEditor framework utilizes a "mutual guided training" approach to enable high-fidelity 3D scene manipulation.
The system consists of three primary components:
- Conditional NeRF: Trained on the target scene, this module generates synthetic data (image-pose pairs) to inform the GAN about the scene's specific 3D structure.
- Latent Code Adjustor: Using differentiable eigen-decomposition, this component identifies pose-specific directions in StyleGAN's latent space. This allows for the disentanglement of camera movement from stylistic attributes like hair color or facial geometry.
- Hidden Mapper: To handle "out-of-domain" views (e.g., extreme side profiles) where standard StyleGANs often fail, the Hidden Mapper projects novel views into the GAN's internal feature space, ensuring style consistency across all angles.
The workflow involves performing a 2D edit (e.g., via text prompts or manual sliders), generating stylized multi-view images through the augmented StyleGAN, and finally fine-tuning the NeRF to "bake" these edits into a consistent 3D representation.
Datasets
The authors evaluate NeRFEditor using two primary benchmarks:
- FaceScape: A large-scale public dataset of 3D face scans featuring various expressions.
- TIFace (Tiny-scale Indoor Face collection): A new dataset introduced in this work, featuring subjects in realistic indoor settings with diverse hairstyles. This provides a more rigorous and realistic test for 3D editing than existing benchmarks.
Results and Discussion
NeRFEditor demonstrates significant improvements in both efficiency and quality over state-of-the-art baselines like CLIP-NeRF and EG3D:
- Efficiency: The pipeline is highly optimized; training the adjustor and fine-tuning the NeRF takes approximately 4 minutes on a single RTX 3070 GPU.
- Identity Preservation: The method achieves superior ID Scores, ensuring the subject's identity remains recognizable even after significant stylistic changes.
- Consistency: The resulting 3D scenes exhibit smooth transitions without the flickering or floating artifacts typical of previous 2D-to-3D editing methods.
- Versatility: The framework supports a wide range of edits, from facial expressions and makeup to class-agnostic style transfers (e.g., applying artistic painting styles to non-facial objects like cars).
Conclusion
NeRFEditor successfully integrates the strengths of 2D GANs and 3D NeRFs through a differentiable decomposition approach. While currently most effective in domains with high-quality pre-trained GANs, it offers a versatile and efficient path forward for high-fidelity 3D content creation. Future work will look toward expanding the domain applicability and handling more extreme geometric deformations.
Summary updated