TL;DR
This paper introduces the first method for 3D-aware multi-class image-to-image (I2I) translation, enabling consistent video generation across different object categories. By decoupling the process into a 3D-aware generative model and a specialized translation step, they achieve impressive view-consistency, overcoming limitations of existing 2D I2I methods.
What Problem Does This Paper Address?
Generating realistic and consistent 3D scenes from images is a challenging task. While recent advances in 3D-aware Generative Adversarial Networks (GANs) and Neural Radiance Fields (NeRFs) have shown great promise for synthesizing high-quality, view-consistent 3D images, applying these to image-to-image translation (e.g., transforming a cat into a dog) has been largely unexplored. Existing 2D I2I methods, when applied to 3D scenes, often result in unrealistic shape and identity changes across different viewpoints, leading to jarring and inconsistent video outputs. This paper aims to bridge this gap by enabling 3D-consistent multi-class I2I translation, where the output remains coherent even when viewed from various angles.
Methods
The authors propose a two-step approach:
- Multi-class 3D-aware Generative Model: They extend the StyleNeRF architecture to handle multiple classes. A key innovation here is an effective training strategy that leverages pre-trained unconditional StyleNeRF models to initialize the multi-class version, preventing training from scratch which often leads to mode collapse. This step ensures the generative model can produce view-consistent 3D scenes for different classes.
- 3D-aware I2I Translation System: Building upon the trained multi-class generative model, they develop a translation system. To address view-consistency issues that still arise, they introduce several novel techniques:
- U-net-like Adaptor: A U-net architecture is used to bridge the encoder and decoder, preserving spatial structure and improving consistency.
- Hierarchical Representation Constraint: This technique encourages consistency by enforcing that intermediate representations within the generator maintain a coherent structure.
- Relative Regularization Loss: This loss function ensures that neighboring patches in the feature space maintain consistent differences, further enhancing view-consistency.
Datasets
The proposed method is trained and evaluated on two datasets:
- Animal Faces (AFHQ): Contains images of animals across three classes (cats, dogs, wildlife).
- CelebA-HQ: Contains human faces, with gender used as the class attribute.
Key Insights and Ideas
- Decoupling is Key: Separating the problem into a 3D-aware generation step and a translation step is crucial for achieving 3D consistency.
- Leveraging Pre-training: Initializing multi-class 3D-aware models with pre-trained unconditional models significantly improves training stability and performance.
- Multi-faceted Consistency: Achieving true 3D-aware I2I translation requires a combination of architectural designs (U-net adaptor) and novel loss functions (hierarchical representation and relative regularization) to combat view-inconsistency.
- Beyond 2D Limitations: Standard 2D I2I methods fundamentally struggle with the inherent 3D structure of scenes, making them unsuitable for generating consistent multi-view outputs.
Key Results
- Quantitative: The proposed method achieves state-of-the-art performance in temporal consistency (TC) on both AFHQ and CelebA-HQ datasets, outperforming baselines like StarGANv2. For instance, they report a TC of 3.743 on CelebA-HQ, significantly better than StarGANv2's 10.250. They also show competitive FID and LPIPS scores.
- Qualitative: The method successfully generates high-quality, view-consistent images across different classes. Visual results demonstrate that it preserves the input pose and style while transforming the object's class, overcoming the unrealistic shape/identity changes seen in 2D I2I methods. User studies also indicate a strong preference for their method's view-consistency.
Limitations and Future Work
- Limitations: While the method significantly improves view-consistency, it's noted that directly using predicted 3D location and viewing direction (in supplementary material) can lead to lower quality images compared to using the input ones. The paper also acknowledges that existing 2D I2I methods like StarGANv2 can sometimes achieve better FID scores, though at the cost of consistency.
- Future Work: The paper suggests exploring unconditional 3D-aware I2I translation and demonstrates its potential. Further work could involve more complex scene translations and exploring the generation of never-seen class embeddings, which their model shows promise in handling.
Summary updated