Abstract
Adapting 3D-aware Generative Adversarial Networks (GANs) to artistic domains presents significant challenges due to "geometric ambiguity"—the discrepancy between 2D artistic expression and 3D physical reality. We present Dr.3D, a framework designed to overcome these hurdles by introducing a deformation-aware architecture and an alternating training strategy. By leveraging geometric priors and residual feature injection, Dr.3D enables high-quality 3D reconstruction and multi-view consistent editing for diverse artistic styles, including anime, ukiyo-e, and historical portraits.
Introduction
Recent advancements in 3D-aware GANs, such as EG3D, have demonstrated the ability to generate high-fidelity, view-consistent human faces. These models typically rely on the assumption that the training data consists of 2D projections of a consistent 3D scene. However, artistic drawings—ranging from classical oil paintings to modern anime—often intentionally deviate from strict perspective and anatomical accuracy.
This "geometric ambiguity" poses a major obstacle for standard 3D GANs. When applied to non-photorealistic art, these models often produce "cardboard cutout" effects or distorted geometries because the 2D lines do not map perfectly to a standard 3D head model. Furthermore, the lack of ground-truth 3D datasets for artistic styles makes supervised learning impossible. Dr.3D addresses these issues through a specialized architecture and a novel adaptation process.
Methodology
Dr.3D builds upon the tri-plane representation of the EG3D architecture, introducing three key innovations to handle the unique characteristics of artistic data.
1. Deformation-aware 3D Synthesis
To accommodate the exaggerated features common in art—such as the large eyes of anime characters or the stylized features of caricatures—Dr.3D incorporates a Deformation Network. This network utilizes a specific latent code ($z_d$) to generate spatially-variant residual features. These features are injected into the generator's early layers, allowing the model to "warp" the standard 3D knowledge learned from real photos to fit the specific proportions and local shape variations of the target artistic domain.
2. Alternating Adaptation
Training a 3D GAN requires accurate camera pose information, yet estimating poses for artistic drawings is difficult without a 3D-aware model. Dr.3D solves this "chicken-and-egg" problem through an alternating adaptation strategy. The framework oscillates between updating a pose estimation network (using synthetic data generated by the GAN) and updating the GAN itself (using poses estimated by the network). This bootstrapping process, starting from a photo-pre-trained model, allows the system to gradually refine its understanding of artistic geometry.
3. Geometric Priors
To ensure the generated 3D shapes remain plausible and do not become nonsensical, the authors employ two specific loss functions:
- Depth Similarity Loss: This forces the 3D shape of the drawing to align with the coarse, low-frequency geometry of a standard human face. By using a Gaussian low-pass filter, the model maintains a basic "head shape" while allowing for stylistic variation.
- Normal Smoothness Loss: This prevents the emergence of jagged or unrealistic surfaces, ensuring the 3D manifold remains continuous and clean.
Experiments and Results
The model was pre-trained on the FFHQ dataset (real human faces) and subsequently adapted to four distinct artistic domains: Historical Art, Ukiyo-e, Anime, and Caricature (using the 3DCaricShop dataset).
Quantitative Performance
Dr.3D demonstrates superior performance compared to state-of-the-art models like StyleNeRF and $\pi$-GAN. On the Historical Art dataset, Dr.3D achieved a Fréchet Inception Distance (FID) of 23.42, a significant improvement over the baseline EG3D's 26.95.
Qualitative and Editing Capabilities
Qualitatively, Dr.3D produces significantly more realistic 3D structures. Unlike previous methods that often resulted in flat images rotating in space, Dr.3D's outputs exhibit genuine depth and volume. This 3D awareness enables robust semantic editing; users can take a single 2D drawing, invert it into the model's latent space, and modify attributes such as expression, gender, or hairstyle while maintaining consistency across all viewing angles.
Conclusion and Future Work
Dr.3D successfully bridges the gap between 2D artistic creativity and 3D generative modeling. By accounting for geometric deformation and utilizing alternating adaptation, it provides a robust framework for stylizing 3D GANs. While the model currently faces challenges with non-geometric backgrounds and extreme abstractions, future work involving decoupled foreground-background generation and expansion to non-human subjects promises to further enhance the versatility of 3D-aware artistic synthesis.
Summary updated