3D-Aware Multi-Class Image-to-Image Translation with NeRFs

Senmao Li, Joost van de Weijer, Yaxing Wang, Fahad Shahbaz Khan, Meiqin Liu, Jian Yang

TL;DR

This paper introduces the first method for 3D-aware multi-class image-to-image (I2I) translation, enabling consistent video generation across different object categories. By decoupling the process into a 3D-aware generative model and a specialized translation step, they achieve impressive view-consistency, overcoming limitations of existing 2D I2I methods.

What Problem Does This Paper Address?

Generating realistic and consistent 3D scenes from images is a challenging task. While recent advances in 3D-aware Generative Adversarial Networks (GANs) and Neural Radiance Fields (NeRFs) have shown great promise for synthesizing high-quality, view-consistent 3D images, applying these to image-to-image translation (e.g., transforming a cat into a dog) has been largely unexplored. Existing 2D I2I methods, when applied to 3D scenes, often result in unrealistic shape and identity changes across different viewpoints, leading to jarring and inconsistent video outputs. This paper aims to bridge this gap by enabling 3D-consistent multi-class I2I translation, where the output remains coherent even when viewed from various angles.

Methods

The authors propose a two-step approach:

  1. Multi-class 3D-aware Generative Model: They extend the StyleNeRF architecture to handle multiple classes. A key innovation here is an effective training strategy that leverages pre-trained unconditional StyleNeRF models to initialize the multi-class version, preventing training from scratch which often leads to mode collapse. This step ensures the generative model can produce view-consistent 3D scenes for different classes.
  2. 3D-aware I2I Translation System: Building upon the trained multi-class generative model, they develop a translation system. To address view-consistency issues that still arise, they introduce several novel techniques:
  3. U-net-like Adaptor: A U-net architecture is used to bridge the encoder and decoder, preserving spatial structure and improving consistency.
  4. Hierarchical Representation Constraint: This technique encourages consistency by enforcing that intermediate representations within the generator maintain a coherent structure.
  5. Relative Regularization Loss: This loss function ensures that neighboring patches in the feature space maintain consistent differences, further enhancing view-consistency.

Datasets

The proposed method is trained and evaluated on two datasets:

  • Animal Faces (AFHQ): Contains images of animals across three classes (cats, dogs, wildlife).
  • CelebA-HQ: Contains human faces, with gender used as the class attribute.

Key Insights and Ideas

  • Decoupling is Key: Separating the problem into a 3D-aware generation step and a translation step is crucial for achieving 3D consistency.
  • Leveraging Pre-training: Initializing multi-class 3D-aware models with pre-trained unconditional models significantly improves training stability and performance.
  • Multi-faceted Consistency: Achieving true 3D-aware I2I translation requires a combination of architectural designs (U-net adaptor) and novel loss functions (hierarchical representation and relative regularization) to combat view-inconsistency.
  • Beyond 2D Limitations: Standard 2D I2I methods fundamentally struggle with the inherent 3D structure of scenes, making them unsuitable for generating consistent multi-view outputs.

Key Results

  • Quantitative: The proposed method achieves state-of-the-art performance in temporal consistency (TC) on both AFHQ and CelebA-HQ datasets, outperforming baselines like StarGANv2. For instance, they report a TC of 3.743 on CelebA-HQ, significantly better than StarGANv2's 10.250. They also show competitive FID and LPIPS scores.
  • Qualitative: The method successfully generates high-quality, view-consistent images across different classes. Visual results demonstrate that it preserves the input pose and style while transforming the object's class, overcoming the unrealistic shape/identity changes seen in 2D I2I methods. User studies also indicate a strong preference for their method's view-consistency.

Limitations and Future Work

  • Limitations: While the method significantly improves view-consistency, it's noted that directly using predicted 3D location and viewing direction (in supplementary material) can lead to lower quality images compared to using the input ones. The paper also acknowledges that existing 2D I2I methods like StarGANv2 can sometimes achieve better FID scores, though at the cost of consistency.
  • Future Work: The paper suggests exploring unconditional 3D-aware I2I translation and demonstrates its potential. Further work could involve more complex scene translations and exploring the generation of never-seen class embeddings, which their model shows promise in handling.

Summary updated

Abstract

Recent advances in 3D-aware generative models (3D-aware GANs) combined with Neural Radiance Fields (NeRF) have achieved impressive results. However no prior works investigate 3D-aware GANs for 3D consistent multiclass image-to-image (3D-aware 121) translation. Naively using 2D-121 translation methods suffers from unrealistic shape/identity change. To perform 3D-aware multiclass 121 translation, we decouple this learning process into a multiclass 3D-aware GAN step and a 3D-aware 121 translation step. In the first step, we propose two novel techniques: a new conditional architecture and an effective training strategy. In the second step, based on the well-trained multiclass 3D-aware GAN architecture, that preserves view-consistency, we construct a 3D-aware 121 translation system. To further reduce the view-consistency problems, we propose several new techniques, including a U-net-like adaptor network design, a hierarchical representation constrain and a relative regularization loss. In exten-sive experiments on two datasets, quantitative and qualitative results demonstrate that we successfully perform 3D-aware 121 translation with multi-view consistency. Code is available in 3DI2I.
BibTeX
@inproceedings{li2023d,
  author = {Senmao Li and Joost van de Weijer and Yaxing Wang and Fahad Shahbaz Khan and Meiqin Liu and Jian Yang},
  title = {{3D-Aware Multi-Class Image-to-Image Translation with NeRFs}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=3so3jnL9EA},
}
Figure