TL;DR
This paper introduces the first framework for 3D-aware multi-class image-to-image (I2I) translation. By decoupling the task into a multi-class 3D-aware GAN step and a specialized translation step using Neural Radiance Fields (NeRFs), the authors achieve high-quality style transfers that remain geometrically consistent across different camera viewpoints.
What Problem Does This Paper Address?
Traditional image-to-image translation methods (like StarGANv2) operate in 2D. While they are excellent at changing a person's gender or turning a dog into a cat in a single image, they fail miserably when applied to video or multi-view data. Because these models lack an understanding of 3D geometry, the "identity" or shape of the translated object often shifts unrealistically as the camera moves—a phenomenon known as view-inconsistency.
Existing 3D-aware generative models (3D GANs) have made strides in synthesizing consistent 3D scenes, but they are typically unconditional (generating random samples) or limited to a single class. This paper bridges the gap, enabling the translation of a source 3D scene into a target class while preserving the original pose and underlying structure.
Methods
The authors propose a decoupled two-step learning process:
- Multi-class 3D-aware GAN: They extend the StyleNeRF architecture to handle multiple classes. A key innovation here is the use of two separate mapping networks: one for class-agnostic features and another for class-specific styles. To ensure stable training, they initialize this model using weights from a pre-trained unconditional StyleNeRF.
- 3D-aware I2I Translation: Once the generative model is trained, they build a translation system. They repurpose a pre-trained discriminator as an encoder ($E$) to extract features from input images. To bridge the gap between this encoder and the generator ($G$), they introduce a U-net-like adaptor ($A$).
To further enforce consistency, the authors introduce three specific technical designs:
- U-net-like Adaptor: Unlike previous ResNet-based adaptors, the U-net structure better preserves the spatial structure of the input features.
- Hierarchical Representation Constraint: This forces the intermediate layers of the generator to produce similar features whether they are driven by the original NeRF path or the translated encoder path.
- Relative Regularization Loss: A novel loss function that ensures the relationship between neighboring patches in the latent space remains consistent, further smoothing out view-dependent artifacts.
Datasets
The model was evaluated on two standard high-resolution benchmarks:
- CelebA-HQ: Used for human face translation, specifically between "male" and "female" classes (~28k images total).
- AFHQ (Animal Faces High Quality): Used for translating between "cat," "dog," and "wildlife" classes (~15k images total).
All images were resized to $256 \times 256$ for the main experiments, with some high-resolution results generated at $1024 \times 1024$.
Key Insights and Ideas
- Decoupling is Essential: Trying to learn 3D geometry and multi-class translation simultaneously is too complex. By training a robust 3D generator first, the translation task becomes a matter of "aligning" the encoder to a known-good 3D latent space.
- Initialization as a Stabilizer: The authors found that training a multi-class 3D GAN from scratch often leads to mode collapse. Transferring knowledge from an unconditional model acts as a critical regularizer.
- The Temporal Consistency (TC) Metric: The authors propose a new metric, $TC = TL / vLPIPS$, where $TL$ is temporal loss (smoothness) and $vLPIPS$ measures diversity across a video. This prevents "cheating" where a model achieves high consistency by simply outputting the same static image regardless of the viewpoint.
Key Results
The proposed method significantly outperforms 2D baselines in maintaining 3D structure.
- Quantitative: On the CelebA-HQ dataset, the method achieved a Temporal Consistency (TC) score of 3.743, vastly superior to StarGANv2's 10.250. Lower is better, indicating much smoother transitions across viewpoints.
- Qualitative: As seen in Figure 4, 2D methods like StarGANv2 suffer from "identity drift" and geometrically impossible features (like warping ears) during viewpoint changes. The proposed method maintains a stable 3D head pose and consistent features.
The model also supports smooth interpolation between classes, allowing for "hybrid" generations that still respect 3D constraints.
Limitations and Future Work
While the paper demonstrates impressive results, a few areas remain for exploration:
- Data Efficiency: The method currently requires large datasets (like AFHQ) to train the base 3D GAN. Adapting this to few-shot scenarios would be a significant step forward.
- Scene Complexity: The experiments focus on centered objects (faces). Extending this to complex, cluttered 3D scenes with multiple objects would be a major challenge for current NeRF-based GAN architectures.
- Inference Speed: NeRF-based rendering is computationally expensive compared to pure 2D convolutions, which might limit real-time applications without further optimization.
Summary updated