3D-Aware Multi-Class Image-to-Image Translation with NeRFs

Senmao Li, Joost van de Weijer, Yaxing Wang, Fahad Shahbaz Khan, Meiqin Liu, Jian Yang

TL;DR

This paper introduces the first framework for 3D-aware multi-class image-to-image (I2I) translation. By decoupling the task into a multi-class 3D-aware GAN step and a specialized translation step using Neural Radiance Fields (NeRFs), the authors achieve high-quality style transfers that remain geometrically consistent across different camera viewpoints.

What Problem Does This Paper Address?

Traditional image-to-image translation methods (like StarGANv2) operate in 2D. While they are excellent at changing a person's gender or turning a dog into a cat in a single image, they fail miserably when applied to video or multi-view data. Because these models lack an understanding of 3D geometry, the "identity" or shape of the translated object often shifts unrealistically as the camera moves—a phenomenon known as view-inconsistency.

Existing 3D-aware generative models (3D GANs) have made strides in synthesizing consistent 3D scenes, but they are typically unconditional (generating random samples) or limited to a single class. This paper bridges the gap, enabling the translation of a source 3D scene into a target class while preserving the original pose and underlying structure.

Methods

The authors propose a decoupled two-step learning process:

  1. Multi-class 3D-aware GAN: They extend the StyleNeRF architecture to handle multiple classes. A key innovation here is the use of two separate mapping networks: one for class-agnostic features and another for class-specific styles. To ensure stable training, they initialize this model using weights from a pre-trained unconditional StyleNeRF.
  2. 3D-aware I2I Translation: Once the generative model is trained, they build a translation system. They repurpose a pre-trained discriminator as an encoder ($E$) to extract features from input images. To bridge the gap between this encoder and the generator ($G$), they introduce a U-net-like adaptor ($A$).
Overview of the proposed two-step method, including the training of the multi-class StyleNeRF and the subsequent I2I translation architecture.

To further enforce consistency, the authors introduce three specific technical designs:

  • U-net-like Adaptor: Unlike previous ResNet-based adaptors, the U-net structure better preserves the spatial structure of the input features.
  • Hierarchical Representation Constraint: This forces the intermediate layers of the generator to produce similar features whether they are driven by the original NeRF path or the translated encoder path.
  • Relative Regularization Loss: A novel loss function that ensures the relationship between neighboring patches in the latent space remains consistent, further smoothing out view-dependent artifacts.

Datasets

The model was evaluated on two standard high-resolution benchmarks:

  • CelebA-HQ: Used for human face translation, specifically between "male" and "female" classes (~28k images total).
  • AFHQ (Animal Faces High Quality): Used for translating between "cat," "dog," and "wildlife" classes (~15k images total).

All images were resized to $256 \times 256$ for the main experiments, with some high-resolution results generated at $1024 \times 1024$.

Key Insights and Ideas

  • Decoupling is Essential: Trying to learn 3D geometry and multi-class translation simultaneously is too complex. By training a robust 3D generator first, the translation task becomes a matter of "aligning" the encoder to a known-good 3D latent space.
  • Initialization as a Stabilizer: The authors found that training a multi-class 3D GAN from scratch often leads to mode collapse. Transferring knowledge from an unconditional model acts as a critical regularizer.
  • The Temporal Consistency (TC) Metric: The authors propose a new metric, $TC = TL / vLPIPS$, where $TL$ is temporal loss (smoothness) and $vLPIPS$ measures diversity across a video. This prevents "cheating" where a model achieves high consistency by simply outputting the same static image regardless of the viewpoint.

Key Results

The proposed method significantly outperforms 2D baselines in maintaining 3D structure.

  • Quantitative: On the CelebA-HQ dataset, the method achieved a Temporal Consistency (TC) score of 3.743, vastly superior to StarGANv2's 10.250. Lower is better, indicating much smoother transitions across viewpoints.
  • Qualitative: As seen in Figure 4, 2D methods like StarGANv2 suffer from "identity drift" and geometrically impossible features (like warping ears) during viewpoint changes. The proposed method maintains a stable 3D head pose and consistent features.
Comparison between StarGANv2 and the proposed method. Note how StarGANv2 underestimates viewpoint changes and produces unrealistic geometric artifacts.

The model also supports smooth interpolation between classes, allowing for "hybrid" generations that still respect 3D constraints.

Interpolation results showing a smooth transition between dog and wildlife classes while maintaining 3D consistency.

Limitations and Future Work

While the paper demonstrates impressive results, a few areas remain for exploration:

  • Data Efficiency: The method currently requires large datasets (like AFHQ) to train the base 3D GAN. Adapting this to few-shot scenarios would be a significant step forward.
  • Scene Complexity: The experiments focus on centered objects (faces). Extending this to complex, cluttered 3D scenes with multiple objects would be a major challenge for current NeRF-based GAN architectures.
  • Inference Speed: NeRF-based rendering is computationally expensive compared to pure 2D convolutions, which might limit real-time applications without further optimization.

Summary updated

Abstract

Recent advances in 3D-aware generative models (3D-aware GANs) combined with Neural Radiance Fields (NeRF) have achieved impressive results. However no prior works investigate 3D-aware GANs for 3D consistent multi-class image-to-image (3D-aware I2I) translation. Naively using 2D-I2I translation methods suffers from unrealistic shape/identity change. To perform 3D-aware multi-class I2I translation, we decouple this learning process into a multi-class 3D-aware GAN step and a 3D-aware I2I translation step. In the first step, we propose two novel techniques: a new conditional architecture and an effective training strategy. In the second step, based on the well-trained multi-class 3D-aware GAN architecture, that preserves view-consistency, we construct a 3D-aware I2I translation system. To further reduce the view-consistency problems, we propose several new techniques, including a U-net-like adaptor network design, a hierarchical representation constrain and a relative regularization loss. In extensive experiments on two datasets, quantitative and qualitative results demonstrate that we successfully perform 3D-aware I2I translation with multi-view consistency.
BibTeX
@inproceedings{li2023d,
  author = {Senmao Li and Joost van de Weijer and Yaxing Wang and Fahad Shahbaz Khan and Meiqin Liu and Jian Yang},
  title = {{3D-Aware Multi-Class Image-to-Image Translation with NeRFs}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=3so3jnL9EA},
}
Figure