UV-IDM: Identity-Conditioned Latent Diffusion Model for Face UV-Texture Generation

Hong Li, Yutang Feng, Song Xue, Xuhui Liu, Bohan Zeng, Shanglin Li, Boyu Liu, Jianzhuang Liu, Shumin Han, Baochang Zhang

Abstract

Generating high-fidelity, identity-preserving 3D face textures from single "in-the-wild" images remains a significant challenge due to occlusions like hair and accessories. This paper introduces UV-IDM, a framework based on Latent Diffusion Models (LDMs) that treats texture generation as an identity-conditioned completion task. By utilizing a novel Identity-Conditioned Module (ICM) and a large-scale "bald" dataset (BFM-UV), UV-IDM achieves state-of-the-art results in realism and identity retention, producing textures in just six seconds.

Introduction

3D face reconstruction is a cornerstone of modern digital media, yet generating realistic "skin" textures that accurately represent a specific individual remains difficult. Traditional linear models often yield blurry results, while GAN-based approaches frequently struggle with identity preservation or "bake" occlusions (like hair or glasses) into the generated texture. UV-IDM addresses these issues by leveraging the generative power of diffusion models conditioned on identity-specific features.

Methodology

The core of UV-IDM is a Latent Diffusion Model (LDM) enhanced by two primary innovations:

  1. Identity-Conditioned Module (ICM): This module extracts partial texture data from the input image using a visibility mask. This incomplete texture is fed into the LDM via cross-attention, ensuring the model fills in occluded areas while strictly adhering to the visible identity features.
  2. "Bald" Data Pipeline: To prevent the model from incorporating hair or accessories into the skin texture, the researchers used StyleGAN2 to synthetically remove hair from training images. This allows the model to learn the underlying facial structure and skin tone, enabling it to hallucinate skin where hair would normally be.

The BFM-UV Dataset

A major contribution of this work is the BFM-UV dataset, which includes over 80,000 subjects. Each entry consists of an "in-the-wild" image, a synthetically "de-haired" version, and a ground-truth UV texture map based on the Basel Face Model (BFM). This scale and variety allow the model to generalize across diverse facial types and lighting conditions.

Results and Discussion

UV-IDM demonstrates significant improvements over existing state-of-the-art methods like HRN and OSTeC. Quantitatively, it achieves lower FID scores (indicating higher visual realism) and higher CSIM scores (indicating better identity retention). Qualitatively, the model produces textures with high-frequency details, such as pores and subtle skin variations, that were previously unattainable with linear models.

Comparison of input images, processed 'bald' versions, generated UV texture maps, and final 3D face renders across four different subjects.

Furthermore, the system is highly efficient, generating a complete texture in approximately six seconds on a P40 GPU—a massive speedup compared to iterative GAN-based methods that can take several minutes.

Conclusion

UV-IDM represents a significant step forward in 3D face texture generation. By framing the problem as a texture-completion task and utilizing a robust identity-conditioning mechanism, it overcomes common hurdles related to occlusions and identity loss. Future work will focus on addressing dataset biases and exploring 3D-aware architectures for even greater structural consistency.

Summary updated

Abstract

3D face reconstruction aims at generating high-fidelity 3D face shapes and textures from single-view or multi-view images. However, current prevailing facial texture generation methods generally suffer from low-quality texture, identity information loss, and inadequate handling of occlusions. To solve these problems, we introduce an Identity-Conditioned Latent Diffusion Model for face UV-texture generation (UV-IDM) to generate photo-realistic textures based on the Basel Face Model (BFM). UV-IDM leverages the powerful texture generation capacity of a latent diffusion model (LDM) to obtain detailed facial textures. To preserve the identity during the reconstruction procedure, we design an identity-conditioned module that can utilize any in-the-wild image as a robust condition for the LDM to guide texture generation. UV-IDM can be easily adapted to different BFM-based methods as a high-fidelity texture generator. Furthermore, in light of the limited accessibility of most existing UV-texture datasets, we build a large-scale and publicly available UV-texture dataset based on BFM, termed BFM-UV. Extensive experiments show that our UV-IDM can generate high-fidelity textures in 3D face reconstruction within seconds while maintaining image consistency, bringing new state-of-the-art performance in facial texture generation.
BibTeX
@inproceedings{li2024uv,
  author = {Hong Li and Yutang Feng and Song Xue and Xuhui Liu and Bohan Zeng and Shanglin Li and Boyu Liu and Jianzhuang Liu and Shumin Han and Baochang Zhang},
  title = {{UV-IDM: Identity-Conditioned Latent Diffusion Model for Face UV-Texture Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01007},
}

Loading related content...

Figure