TL;DR
Generating high-quality 3D stylized portraits traditionally requires either expensive multi-view dataset collection or sluggish per-prompt optimization. This paper introduces an efficient framework that uses text prompts to synthesize consistent, high-fidelity 3D stylized portraits in roughly three minutes. By coupling an optimized 2D diffusion editor with an "Image-to-Triplane" (I2T) module replacing EG3D's generator, the model escapes photorealistic data constraints while preserving 3D structural consistency.
What Problem Does This Paper Address?
State-of-the-art 3D-aware GANs (such as EG3D) synthesize photo-consistent portraits with high fidelity, but they are constrained by their training distributions—typically photorealistic datasets like FFHQ. Generating out-of-distribution, stylized portraits (e.g., Disney characters, pixel art, or sculptures) is notoriously difficult:
- Retraining 3D GANs on multi-style 3D data is computationally prohibitive and requires scarce 3D assets.
- 3D GAN inversion fails to reproduce out-of-distribution artistic styles or destroys geometry.
- Applying 2D style transfer tools per viewpoint destroys multi-view consistency.
- Score distillation / Text-to-3D approaches (e.g., DreamFusion) are slow and often produce blurry or artifact-heavy facial shapes.
The authors ask: How can we synthesize high-quality, text-specified 3D portraits quickly without collecting new 3D stylized datasets?
Methods
[ EG3D Prior ] ---> Multi-view Renders ---> [ Optimized Ip2p ] ---> Few-Shot Dataset (Ds)
|
[ Front View ] ---> [ Style Encoder E ] ---> Spatially Expanded Latent (ws) |
| v
[ SMLs Layers ] ---> [ Triplane ] ---> [ EG3D Renderer ]
The pipeline operates in two core stages:
- Few-Shot Stylized Dataset Construction:
- A single latent identity is sampled from a pre-trained EG3D model and rendered across grid angles ($10 \times 10 = 100$ viewpoints within $\pm 30^\circ$ pitch and yaw).
- The multi-view images are edited with InstructPix2Pix (Ip2p) guided by a prompt $t^* = \{t, t_d, t_n\}$ containing target, decorative, and negative prompts.
- Inference Optimization: Pure random noise causes Ip2p outputs to fluctuate wildly across camera angles. To resolve this multi-view misalignment, the authors adopt an SDEdit-inspired scheme: standard Gaussian noise is replaced with latent features of the rendered image noised to step $\tau$ ($n^* = \text{Add}(E(I), n, \tau)$), anchoring the style and identity across views.
- Image-to-Triplane (I2T) Network & Fast Fine-Tuning:
- Rather than fine-tuning the entire EG3D backbone (which risks degrading 3D priors), the authors replace only EG3D's triplane generator with an Image-to-Triplane (I2T) network while freezing the volume renderer.
- The I2T module maps a front-view image to an expanded spatial latent code $w_s \in \mathbb{R}^{c \times k \times k}$ ($k=32$), which feeds into 7 StyleGAN modulation layers (SMLs) to predict triplane features.
- Pre-training & Adaptation: I2T is first pre-trained for 100k iterations on EG3D outputs with an $L_1$ triplane loss. It is then fine-tuned on the few-shot set $D_s$ using image reconstruction loss ($L_1 + \text{LPIPS}$) combined with a density field smoothness regularization ($L_{dr}$) to avoid hollow or noisy geometries.
Datasets
- Pre-training Base: Pre-trained EG3D checkpoints trained on FFHQ (Flickr-Faces-HQ).
- Per-Prompt Dataset: Dynamically synthesized few-shot datasets ($D_s$), each consisting of ~100 multi-view stylized images per target text prompt generated using EG3D and optimized Ip2p. No external 3D stylized dataset is required.
Key Insights and Ideas
- Partially-noised latents stabilize multi-view diffusion: Standard diffusion editing from independent Gaussian noise yields conflicting visual details across perspectives. Blending the rendering’s latent feature with noise acts as an identity anchor that dramatically reduces viewpoint divergence.
- Spatially expanded latent representation: Compressing stylized portraits into a flat 1D latent vector restricts detail. Expanding $w_s$ into a 2D spatial feature map ($32 \times 32$) gives the network sufficient expressiveness to preserve non-photorealistic textures and silhouettes.
- Decoupled triplane adaptation: Freezing the rendering neural field and only adapting the triplane generator allows rapid convergence (~3 minutes) while keeping density fields smooth and artifact-free.
- Latent style interpolation: Linearly interpolating between two stylized representations in the $w_s$ space enables smooth transitions and cross-style mixing.
Key Results
- Fast Training: Fine-tuning converges in $\sim$3 minutes on a single NVIDIA A100 GPU (compared to $\sim$40 minutes for DreamFusion and $\sim$10 minutes for HFGI3D).
- Quantitative Performance:
- CLIP Score: Achieves 0.332, outperforming DreamFusion (0.304), HFGI3D (0.310), and 3DGAN-Inv (0.229).
- User Study (5-point scale): Scores 4.05 in Text-Image Similarity, 4.27 in Image Quality, and 4.34 in 3D Consistency—ranking highest across all dimensions against baselines.
- Ablation Validation: Omitting I2T pre-training leads to mode collapse (CLIP drops to 0.113), while removing the spatial dimension in $w_s$ drops CLIP to 0.282 and results in blurry outputs.
Limitations and Future Work
- Prior Dependencies: The method inherits biases and failure modes from InstructPix2Pix; slight stylistic mismatches across viewpoints can still bleed into the final 3D reconstruction.
- Extreme Non-Human Shapes: Styles requiring substantial geometric deviation from human anatomy (e.g., rigid helmets like Iron Man or Stormtrooper armor) lead to degradation because 2D diffusion produces inconsistent geometric outlines across viewpoints.
- Prompt Engineering: While default decorative/negative prompts perform well across varied styles, fine-tuning prompt weights manually is still needed for fringe artistic styles.
Summary updated