NeRFEditor: Differentiable Style Decomposition for 3D Scene Editing

Chunyi Sun, Yanbin Liu, Junlin Han, Stephen Gould

Introduction

Neural Radiance Fields (NeRF) have revolutionized 3D scene reconstruction, yet they remain difficult to edit due to their implicit nature. Conversely, 2D Generative Adversarial Networks (GANs), such as StyleGAN, offer intuitive latent spaces for editing but lack 3D consistency across varying viewpoints. NeRFEditor bridges this gap by introducing a framework that combines the high-fidelity editability of 2D GANs with the structural consistency of NeRF, enabling 360-degree scene manipulation.

Methodology

The NeRFEditor framework utilizes a "mutual guided training" approach to enable high-fidelity 3D scene manipulation.

Overview of the NeRFEditor framework, showing (a) NeRF rendering, (b) StyleGAN generation, and (c) the mutual guided training process.

The system consists of three primary components:

  1. Conditional NeRF: Trained on the target scene, this module generates synthetic data (image-pose pairs) to inform the GAN about the scene's specific 3D structure.
  2. Latent Code Adjustor: Using differentiable eigen-decomposition, this component identifies pose-specific directions in StyleGAN's latent space. This allows for the disentanglement of camera movement from stylistic attributes like hair color or facial geometry.
  3. Hidden Mapper: To handle "out-of-domain" views (e.g., extreme side profiles) where standard StyleGANs often fail, the Hidden Mapper projects novel views into the GAN's internal feature space, ensuring style consistency across all angles.

The workflow involves performing a 2D edit (e.g., via text prompts or manual sliders), generating stylized multi-view images through the augmented StyleGAN, and finally fine-tuning the NeRF to "bake" these edits into a consistent 3D representation.

Datasets

The authors evaluate NeRFEditor using two primary benchmarks:

  • FaceScape: A large-scale public dataset of 3D face scans featuring various expressions.
  • TIFace (Tiny-scale Indoor Face collection): A new dataset introduced in this work, featuring subjects in realistic indoor settings with diverse hairstyles. This provides a more rigorous and realistic test for 3D editing than existing benchmarks.

Results and Discussion

NeRFEditor demonstrates significant improvements in both efficiency and quality over state-of-the-art baselines like CLIP-NeRF and EG3D:

  • Efficiency: The pipeline is highly optimized; training the adjustor and fine-tuning the NeRF takes approximately 4 minutes on a single RTX 3070 GPU.
  • Identity Preservation: The method achieves superior ID Scores, ensuring the subject's identity remains recognizable even after significant stylistic changes.
  • Consistency: The resulting 3D scenes exhibit smooth transitions without the flickering or floating artifacts typical of previous 2D-to-3D editing methods.
  • Versatility: The framework supports a wide range of edits, from facial expressions and makeup to class-agnostic style transfers (e.g., applying artistic painting styles to non-facial objects like cars).

Conclusion

NeRFEditor successfully integrates the strengths of 2D GANs and 3D NeRFs through a differentiable decomposition approach. While currently most effective in domains with high-quality pre-trained GANs, it offers a versatile and efficient path forward for high-fidelity 3D content creation. Future work will look toward expanding the domain applicability and handling more extreme geometric deformations.

Summary updated

Abstract

We present NeRFEditor, an efficient learning framework for 3D scene editing, which takes a video as input and outputs a high-quality, identity-preserving stylized 3D scene. Our goal is to bridge the gap between 2D and 3D editing, catering to a wide array of creative modifications such as reference-guided alterations, text-based prompts, and user interactions. We achieve this by encouraging a pre-trained StyleGAN model and a NeRF model to learn mutually consistent renderings. Specifically, we use NeRF to generate numerous (image, camera pose)-pairs to train an adjustor module, which adapts the StyleGAN latent code for generating high-fidelity stylized images from any given viewing angle. To extrapolate edits to novel views, i.e., those not seen by StyleGAN pre-training, while maintaining 360° consistency, we propose a second self-supervised module that maps these views into the hidden space of StyleGAN. Together these two modules produce sufficient guidance for NeRF to learn consistent stylization effects across the full range of views. Experiments show that NeRFEditor outperforms prior work on benchmark and real-world scenes with better editability, fidelity, and identity preservation.
BibTeX
@inproceedings{sun2024nerfeditor,
  author = {Chunyi Sun and Yanbin Liu and Junlin Han and Stephen Gould},
  title = {{NeRFEditor: Differentiable Style Decomposition for 3D Scene Editing}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00714},
}
Figure