Abstract
Automatic 3D facial texture generation has gained significant interest recently. Existing approaches may not support the traditional physically based rendering pipeline or rely on 3D data captured by Light Stage. Our key contribution is a progressive latent space refinement approach that can bootstrap from 3D Morphable Models (3DMMs)-based texture maps generated from facial images to generate high-quality and diverse PBR textures, including albedo, normal, and roughness. It starts with enhancing Generative Adversarial Networks (GANs) for text-guided and diverse texture generation. To this end, we design a self-supervised paradigm to overcome the reliance on ground truth 3D textures and train the generative model with only entangled texture maps. Besides, we foster mutual enhancement between GANs and Score Distillation Sampling (SDS). SDS boosts GANs with more generative modes, while GANs promote more efficient optimization of SDS. Furthermore, we introduce an edge-aware SDS for multi-view consistent facial structure. Experiments demonstrate that our method outperforms existing 3D texture generation methods regarding photo-realistic quality, diversity, and efficiency.
Introduction
Creating realistic 3D avatars often hits a bottleneck at texture generation. While text-to-image AI has advanced rapidly, translating complex prompts like "a man with a prominent knife scar" into high-quality, physically-based rendering (PBR) textures—comprising albedo, normal, and roughness maps—remains a significant challenge. Traditional methods often rely on expensive "Light Stage" 3D scans or suffer from slow optimization times.
This paper introduces PBRGAN, a framework that utilizes a progressive latent-space refinement approach to generate diverse, high-fidelity facial textures from text prompts without requiring specialized 3D data. By bootstrapping from 2D facial image datasets, PBRGAN achieves state-of-the-art diversity and quality with significantly faster inference than previous diffusion-based methods.
Methodology
The PBRGAN framework operates through a three-stage pipeline designed to refine the latent space of a generative model progressively:
- Disentanglement: A StyleGAN2-based generator is trained to produce PBR maps using a self-supervised paradigm. By passing generated maps through a differentiable renderer and comparing them to entangled UV maps from the FFHQ-UV dataset, the model learns to separate lighting from true surface properties.
- Alignment: The GAN’s latent space ($W+$) is aligned with text embeddings using CLIP, enabling fast, feedforward generation for common facial attributes.
- Amalgamation & Expansion: To handle "out-of-distribution" prompts, the GAN is combined with Score Distillation Sampling (SDS) from a pre-trained diffusion model. A novel Edge-Aware SDS (EASDS) utilizes ControlNet and HED edges to ensure structural consistency across multiple views, preventing common issues like the "Janus problem."
Results and Discussion
PBRGAN demonstrates superior performance in both efficiency and quality. For common prompts, textures are generated in less than 0.01 minutes. Even complex prompts requiring full EASDS refinement take approximately 1 minute, which is significantly faster than competitors like Fantasia3D (45 mins) or TANGO (10 mins).
Key insights from the study include:
- Mutual Enhancement: The synergy between GANs and SDS allows the GAN to provide a structured starting point, making SDS optimization faster and more stable, while SDS provides the creative spark to exceed the GAN's training data limitations.
- Self-Supervised PBR: Differentiable rendering allows for high-quality PBR map generation without ground-truth 3D data.
- Structural Priors: EASDS acts as a structural anchor, ensuring that specific features like scars or lip colors remain correctly positioned on the 3D model regardless of the viewing angle.
Conclusion
PBRGAN offers a robust solution for text-driven 3D facial texture generation, outperforming existing methods in speed and photo-realism. While the current framework relies on pre-existing facial geometries, future work aims to explore the joint generation of both accurate facial geometry and textures to create even more cohesive 3D avatars.
Summary updated