FaceRiff: Data-Efficient Unified Text-to-3D Face Asset Synthesis via Geometric Conditioning

Elly Akhoundi, Mathieu Lamarre, Thibault Goyallon, Patrick Anderson, Étienne DanvoyearXiv 2026 · arXiv

TL;DR

FaceRiff is a new framework that generates high-quality, render-ready 3D face assets—including mesh, lighting-independent albedo, and pore-scale displacement—directly from text prompts. By adapting a pretrained Stable Diffusion XL (SDXL) model with parameter-efficient adapters and a clever "geometric conditioning" strategy, it achieves state-of-the-art realism using a fraction of the data required by previous methods.

What Problem Does This Paper Address?

Creating photorealistic digital humans is a massive undertaking. It typically requires a "stack" of three distinct assets: an accurate 3D mesh, a "lighting-independent" albedo (the base color of the skin without shadows or highlights), and a displacement map for fine details like pores and wrinkles.

Currently, creators face a "pick two" dilemma:

  1. High Quality vs. Speed: Score-distillation methods produce great results but take hours per face.
  2. Realism vs. Data: Training a model from scratch to understand 3D facial geometry requires massive, often proprietary, datasets.
  3. Unified Assets: Most generative models focus on just one part of the stack (e.g., just the mesh or just the texture), leading to inconsistencies when you try to combine them.

FaceRiff bridges this gap by treating 3D generation as a 2D image synthesis task within a shared, fixed UV space, allowing it to generate a complete, synchronized asset stack in seconds.

Fig. 1. FaceRiff: text-driven 3D face asset generation. From a text prompt, FaceRiff synthesizes mesh, lighting-independent albedo, and pore-scale displacement in a shared UV space in a feed-forward manner, combining them into the final rendered full head.

Methods

The core innovation of FaceRiff is how it "repurposes" a standard 2D image diffusion model (SDXL) for 3D tasks.

  • Shared UV Space: Instead of generating 3D volumes, FaceRiff generates 2D maps (Position, Albedo, and Displacement) that correspond to a standardized facial template. This ensures that every generated asset "fits" the same head perfectly.
  • Geometric Conditioning: To help the model understand facial anatomy, the authors use a frozen ControlNet fed with UV normal maps. For the mesh, they use a "dataset-mean" normal map as a spatial scaffold. For albedo and displacement, they use the normals derived from the newly generated mesh.
  • Parameter-Efficient Fine-Tuning (PEFT): Rather than retraining the massive SDXL model, they use LoRA adapters. This updates less than 3% of the model's parameters per task, making it much easier to train on limited hardware.
  • Residual Position Maps: To generate the mesh, the model doesn't predict absolute coordinates. Instead, it predicts "offsets" (residuals) from a mean face shape. This makes the learning task much simpler and more stable.
The FaceRiff pipeline uses task-specific LoRA adapters and a frozen ControlNet to synthesize different modalities in a cascaded fashion.

Datasets

The authors trained FaceRiff on a relatively small but high-quality dataset of 2,049 facial scans from approximately 350 identities.

  • VLM-based Tagging: Because raw scans lack descriptive metadata, they used Gemini 2.5 Pro to generate incredibly detailed, structured captions for every scan, covering demographics, skin details, and facial expressions (FACS).
  • Asset Types: Each training sample included 8-bit sRGB albedo, 32-bit displacement maps, and 1024x1024 UV position maps.

Key Insights and Ideas

  • Normals as Scaffolding: The "aha moment" is using normal maps as a spatial anchor. Even a generic "mean" normal map provides enough structural context for the diffusion model to know exactly where the nose, eyes, and mouth should be in UV space.
  • Cascaded Inference: By generating the mesh first and then using its geometry to guide the texture generation, the model achieves "cross-modality alignment." The wrinkles in the displacement map actually line up with the creases in the mesh.
  • Diffusion Priors Transfer: The paper proves that the "knowledge" a model like SDXL has about what a human face looks like in photos can be successfully transferred to non-image data like 3D position and displacement maps.

Key Results

FaceRiff significantly outperforms existing baselines like DreamFace and FaceG2E in both diversity and realism.

  • Perceived Realism: In a blind study with 45 industry professionals (character artists and rendering engineers), FaceRiff's outputs were judged "real" 59.4% of the time, compared to only 8.9% for DreamFace.
  • Diversity: Unlike previous methods that often collapse into a single "average" face for a given prompt, FaceRiff produces distinct identities across different random seeds while remaining faithful to the text description.
  • PBR Readiness: Because the albedo is lighting-independent (no "baked-in" shadows), the assets can be dropped into any game engine (like Unreal or Unity) and relit realistically.
Compared to DreamFace, FaceRiff produces sharper details, lighting-independent textures, and much higher diversity across different seeds.

Limitations and Future Work

  • Error Propagation: Since the texture stages depend on the mesh stage, any significant errors in the generated 3D shape can "leak" into the albedo and displacement maps.
  • Fixed Topology: While using a fixed template ensures consistency, it limits the model's ability to generate extreme variations in head shape or non-human features.
  • Data Constraints: The model is currently limited by the diversity of its 2,049-scan training set. Expanding this to include more diverse ethnicities, ages, and expressions is a clear next step.

Summary updated

Abstract

High-resolution facial datasets with consistent UV-mapped geometry, albedo, and physically based displacement are scarce, yet such assets are essential for photorealistic digital-human rendering. Existing 3D face generation methods are often slow, single-modality, or require large specialized datasets. Meanwhile, large image diffusion models encode strong facial priors but do not natively produce lighting-independent textures, pore-scale displacement, or UV-aligned geometry. We present FaceRiff, a unified, data- and parameter-efficient framework for generating complete 3D face assets (mesh, albedo, and displacement) directly from text in a shared UV space. FaceRiff adapts a pretrained SDXL backbone with task-specific LoRA adapters and uses UV normal maps as geometric conditioning: per-instance normals guide albedo and displacement synthesis, while a dataset-mean normal provides a canonical spatial anchor for mesh generation. Using only 2,049 scans from approximately 350 identities and training on a single GPU, FaceRiff updates less than 3% of the backbone parameters per task and produces render-ready 3D face assets without per-instance optimization. Compared with FaceG2E and DreamFace, FaceRiff produces greater per-prompt diversity while preserving prompt alignment and fixed-topology UV consistency, and additionally generates lighting-independent albedo suitable for PBR relighting and pore-scale displacement absent from prior text-driven baselines. A blind study with 45 industry professionals shows that FaceRiff achieves substantially higher perceived realism than DreamFace, approaching that of real captures. Overall, FaceRiff demonstrates that pretrained diffusion priors can be adapted to unified 3D face asset synthesis under limited data.
BibTeX
@misc{akhoundi2026faceriff,
  author = {Elly Akhoundi and Mathieu Lamarre and Thibault Goyallon and Patrick Anderson and Étienne Danvoye},
  title = {{FaceRiff: Data-Efficient Unified Text-to-3D Face Asset Synthesis via Geometric Conditioning}},
  year = {2026},
  eprint = {2611.21867},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21867},
}
Figure