Abstract
Generating high-fidelity, identity-preserving 3D face textures from single "in-the-wild" images remains a significant challenge due to occlusions like hair and accessories. This paper introduces UV-IDM, a framework based on Latent Diffusion Models (LDMs) that treats texture generation as an identity-conditioned completion task. By utilizing a novel Identity-Conditioned Module (ICM) and a large-scale "bald" dataset (BFM-UV), UV-IDM achieves state-of-the-art results in realism and identity retention, producing textures in just six seconds.
Introduction
3D face reconstruction is a cornerstone of modern digital media, yet generating realistic "skin" textures that accurately represent a specific individual remains difficult. Traditional linear models often yield blurry results, while GAN-based approaches frequently struggle with identity preservation or "bake" occlusions (like hair or glasses) into the generated texture. UV-IDM addresses these issues by leveraging the generative power of diffusion models conditioned on identity-specific features.
Methodology
The core of UV-IDM is a Latent Diffusion Model (LDM) enhanced by two primary innovations:
- Identity-Conditioned Module (ICM): This module extracts partial texture data from the input image using a visibility mask. This incomplete texture is fed into the LDM via cross-attention, ensuring the model fills in occluded areas while strictly adhering to the visible identity features.
- "Bald" Data Pipeline: To prevent the model from incorporating hair or accessories into the skin texture, the researchers used StyleGAN2 to synthetically remove hair from training images. This allows the model to learn the underlying facial structure and skin tone, enabling it to hallucinate skin where hair would normally be.
The BFM-UV Dataset
A major contribution of this work is the BFM-UV dataset, which includes over 80,000 subjects. Each entry consists of an "in-the-wild" image, a synthetically "de-haired" version, and a ground-truth UV texture map based on the Basel Face Model (BFM). This scale and variety allow the model to generalize across diverse facial types and lighting conditions.
Results and Discussion
UV-IDM demonstrates significant improvements over existing state-of-the-art methods like HRN and OSTeC. Quantitatively, it achieves lower FID scores (indicating higher visual realism) and higher CSIM scores (indicating better identity retention). Qualitatively, the model produces textures with high-frequency details, such as pores and subtle skin variations, that were previously unattainable with linear models.
Furthermore, the system is highly efficient, generating a complete texture in approximately six seconds on a P40 GPU—a massive speedup compared to iterative GAN-based methods that can take several minutes.
Conclusion
UV-IDM represents a significant step forward in 3D face texture generation. By framing the problem as a texture-completion task and utilizing a robust identity-conditioning mechanism, it overcomes common hurdles related to occlusions and identity loss. Future work will focus on addressing dataset biases and exploring 3D-aware architectures for even greater structural consistency.
Summary updated