TL;DR
This paper introduces ZIGNERF, a novel method for generating 3D-consistent multi-view images from a single input image without requiring any training on that specific image. It achieves this by combining a 3D-aware generative NeRF model with a zero-shot GAN inversion technique, enabling impressive control over object pose and background.
What Problem Does This Paper Address?
Generating high-quality 3D representations of scenes from single images has been a long-standing challenge. While generative Neural Radiance Fields (NeRFs) have shown great promise in synthesizing realistic 3D scenes from multiple views or learning distributions from unposed images, they typically struggle with creating a 3D representation from a single input image, especially if that image is "out-of-domain" (i.e., not similar to the training data). Existing methods often require extensive fine-tuning on the specific input image, which is computationally expensive and impractical for large-scale applications. This paper aims to bridge this gap by enabling zero-shot 3D GAN inversion for single, out-of-domain images.
Methods
ZIGNERF is built upon two core components:
- 3D-Generation Module: This module is based on the GIRAFFE [27] architecture, which effectively captures the compositional nature of 3D scenes. The authors introduce modifications to the neural renderer and discriminator to improve reconstruction accuracy and image quality. This component is trained to produce photorealistic, 3D-consistent images from latent codes and camera poses.
- 3D-Aware GAN Inverter: This is the novel contribution. It's a specialized inverter designed to map out-of-domain images directly into the latent space of the pre-trained generator. Unlike previous methods that rely on optimization-based GAN inversion and require fine-tuning, ZIGNERF's inverter is trained to perform this mapping in a zero-shot manner. The inverter is trained using synthesized images from the generator, allowing it to learn to extract features and map them to the generator's manifold, regardless of the input image's pose. The training objective for the inverter includes a combination of GAN loss, L1 reconstruction loss, and perceptual losses (SSIM and LPIPS) to ensure both realism and detail preservation.
Datasets
The paper evaluates ZIGNERF on several real-world image datasets:
- Cats
- CelebA [22]
- CelebA-HQ [15]
- AFHQ [5]
- CompCars [45]
Additionally, FFHQ [17] images are used to demonstrate the model's robustness and adaptability to unseen data distributions.
Key Insights and Ideas
- Zero-Shot GAN Inversion for 3D: The core innovation is achieving 3D-aware GAN inversion without requiring any fine-tuning on the input image. This significantly broadens the applicability of 3D generative models to real-world, diverse inputs.
- Disentanglement of Object and Background: The model effectively separates object features from the background, allowing for independent manipulation and control.
- Robust 3D Controllability: ZIGNERF demonstrates strong control over 3D operations like 360-degree rotation, depth translation, and horizontal translation, even for real-world images.
- Improved Neural Renderer: Modifications to the neural renderer, including residual blocks and spectral normalization, enhance training stability and output quality compared to previous approaches.
Key Results
- Quantitative Evaluation: ZIGNERF significantly outperforms the baseline GIRAFFE model in terms of Fréchet Inception Distance (FID) scores across multiple datasets and resolutions for both unconditional and conditional generation. For example, on the CelebA(HQ) 256x256 dataset, ZIGNERF achieves an FID of 14.98, compared to GIRAFFE's 23.14.
- Qualitative Results: The paper showcases impressive qualitative results, demonstrating the generation of multi-view consistent images from single input images across various domains (faces, animals, cars). The visualizations highlight the model's ability to perform 3D operations and disentangle objects from their backgrounds.
Limitations and Future Work
- Limitations: The paper acknowledges that while the model is trained on single-object scenes, it can generate scenes with multiple objects by leveraging multiple decoder segments. However, this has only been empirically explored on one dataset.
- Future Work: The authors suggest extending the proposed method by manipulating the inverted latent code for editing the input image, further enhancing the capabilities of this innovative model. They also believe their pipeline can be generally applied to other existing generative NeRFs.
Summary updated