Free-style and Fast 3D Portrait Synthesis

Tianxiang Ma, Kang Zhao, Jianxin Sun, Yingya Zhang, Jing Dong, Tieniu TanSubmission to ICLR 2024 · Withdrawn

TL;DR

Generating high-quality 3D stylized portraits traditionally requires either expensive multi-view dataset collection or sluggish per-prompt optimization. This paper introduces an efficient framework that uses text prompts to synthesize consistent, high-fidelity 3D stylized portraits in roughly three minutes. By coupling an optimized 2D diffusion editor with an "Image-to-Triplane" (I2T) module replacing EG3D's generator, the model escapes photorealistic data constraints while preserving 3D structural consistency.

What Problem Does This Paper Address?

State-of-the-art 3D-aware GANs (such as EG3D) synthesize photo-consistent portraits with high fidelity, but they are constrained by their training distributions—typically photorealistic datasets like FFHQ. Generating out-of-distribution, stylized portraits (e.g., Disney characters, pixel art, or sculptures) is notoriously difficult:

  • Retraining 3D GANs on multi-style 3D data is computationally prohibitive and requires scarce 3D assets.
  • 3D GAN inversion fails to reproduce out-of-distribution artistic styles or destroys geometry.
  • Applying 2D style transfer tools per viewpoint destroys multi-view consistency.
  • Score distillation / Text-to-3D approaches (e.g., DreamFusion) are slow and often produce blurry or artifact-heavy facial shapes.

The authors ask: How can we synthesize high-quality, text-specified 3D portraits quickly without collecting new 3D stylized datasets?

Methods

[ EG3D Prior ] ---> Multi-view Renders ---> [ Optimized Ip2p ] ---> Few-Shot Dataset (Ds)
                                                                            |
[ Front View ] ---> [ Style Encoder E ] ---> Spatially Expanded Latent (ws) |
                                                     |                      v
                                             [ SMLs Layers ] ---> [ Triplane ] ---> [ EG3D Renderer ]

The pipeline operates in two core stages:

  1. Few-Shot Stylized Dataset Construction:
  2. A single latent identity is sampled from a pre-trained EG3D model and rendered across grid angles ($10 \times 10 = 100$ viewpoints within $\pm 30^\circ$ pitch and yaw).
  3. The multi-view images are edited with InstructPix2Pix (Ip2p) guided by a prompt $t^* = \{t, t_d, t_n\}$ containing target, decorative, and negative prompts.
  4. Inference Optimization: Pure random noise causes Ip2p outputs to fluctuate wildly across camera angles. To resolve this multi-view misalignment, the authors adopt an SDEdit-inspired scheme: standard Gaussian noise is replaced with latent features of the rendered image noised to step $\tau$ ($n^* = \text{Add}(E(I), n, \tau)$), anchoring the style and identity across views.
  5. Image-to-Triplane (I2T) Network & Fast Fine-Tuning:
  6. Rather than fine-tuning the entire EG3D backbone (which risks degrading 3D priors), the authors replace only EG3D's triplane generator with an Image-to-Triplane (I2T) network while freezing the volume renderer.
  7. The I2T module maps a front-view image to an expanded spatial latent code $w_s \in \mathbb{R}^{c \times k \times k}$ ($k=32$), which feeds into 7 StyleGAN modulation layers (SMLs) to predict triplane features.
  8. Pre-training & Adaptation: I2T is first pre-trained for 100k iterations on EG3D outputs with an $L_1$ triplane loss. It is then fine-tuned on the few-shot set $D_s$ using image reconstruction loss ($L_1 + \text{LPIPS}$) combined with a density field smoothness regularization ($L_{dr}$) to avoid hollow or noisy geometries.

Datasets

  • Pre-training Base: Pre-trained EG3D checkpoints trained on FFHQ (Flickr-Faces-HQ).
  • Per-Prompt Dataset: Dynamically synthesized few-shot datasets ($D_s$), each consisting of ~100 multi-view stylized images per target text prompt generated using EG3D and optimized Ip2p. No external 3D stylized dataset is required.

Key Insights and Ideas

  • Partially-noised latents stabilize multi-view diffusion: Standard diffusion editing from independent Gaussian noise yields conflicting visual details across perspectives. Blending the rendering’s latent feature with noise acts as an identity anchor that dramatically reduces viewpoint divergence.
  • Spatially expanded latent representation: Compressing stylized portraits into a flat 1D latent vector restricts detail. Expanding $w_s$ into a 2D spatial feature map ($32 \times 32$) gives the network sufficient expressiveness to preserve non-photorealistic textures and silhouettes.
  • Decoupled triplane adaptation: Freezing the rendering neural field and only adapting the triplane generator allows rapid convergence (~3 minutes) while keeping density fields smooth and artifact-free.
  • Latent style interpolation: Linearly interpolating between two stylized representations in the $w_s$ space enables smooth transitions and cross-style mixing.

Key Results

  • Fast Training: Fine-tuning converges in $\sim$3 minutes on a single NVIDIA A100 GPU (compared to $\sim$40 minutes for DreamFusion and $\sim$10 minutes for HFGI3D).
  • Quantitative Performance:
  • CLIP Score: Achieves 0.332, outperforming DreamFusion (0.304), HFGI3D (0.310), and 3DGAN-Inv (0.229).
  • User Study (5-point scale): Scores 4.05 in Text-Image Similarity, 4.27 in Image Quality, and 4.34 in 3D Consistency—ranking highest across all dimensions against baselines.
  • Ablation Validation: Omitting I2T pre-training leads to mode collapse (CLIP drops to 0.113), while removing the spatial dimension in $w_s$ drops CLIP to 0.282 and results in blurry outputs.

Limitations and Future Work

  • Prior Dependencies: The method inherits biases and failure modes from InstructPix2Pix; slight stylistic mismatches across viewpoints can still bleed into the final 3D reconstruction.
  • Extreme Non-Human Shapes: Styles requiring substantial geometric deviation from human anatomy (e.g., rigid helmets like Iron Man or Stormtrooper armor) lead to degradation because 2D diffusion produces inconsistent geometric outlines across viewpoints.
  • Prompt Engineering: While default decorative/negative prompts perform well across varied styles, fine-tuning prompt weights manually is still needed for fringe artistic styles.

Summary updated

Abstract

Efficiently generating a free-style 3D portrait with high quality and consistency is a promising yet challenging task. The portrait styles generated by most existing methods are usually restricted by their 3D generators, which are learned in specific facial datasets, such as FFHQ. To get a free-style 3D portrait, one can build a large-scale multi-style database to retrain the 3D generator, or use a off-the-shelf tool to do the style translation. However, the former is time-consuming due to data collection and training process, the latter may destroy the multi-view consistency. To tackle this problem, we propose a fast 3D portrait synthesis framework in this paper, which enable one to use text prompts to specify styles. Specifically, for a given portrait style, we first leverage two generative priors, a 3D-aware GAN generator (EG3D) and a text-guided image editor (Ip2p), to quickly construct a few-shot training set, where the inference process of Ip2p is optimized to make editing more stable. Then we replace original triplane generator of EG3D with a Image-to-Triplane (I2T) module for two purposes: 1) getting rid of the style constraints of pre-trained EG3D by fine-tuning I2T on the few-shot dataset; 2) improving training efficiency by fixing all parts of EG3D except I2T. Furthermore, we construct a multi-style and multi-identity 3D portrait database to demonstrate the scalability and generalization of our method. Experimental results show that our method is capable of synthesizing high-quality 3D portraits with specified styles in a few minutes, outperforming the state-of-the-art.
BibTeX
@misc{ma2023free,
  author = {Tianxiang Ma and Kang Zhao and Jianxin Sun and Yingya Zhang and Jing Dong and Tieniu Tan},
  title = {{Free-style and Fast 3D Portrait Synthesis}},
  year = {2023},
  eprint = {2306.15419},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.15419},
}

Loading related content...

Figure