InViTe: Individual Virtual Transfer for Personalized 3D Face Generation System

Mingyu Jang, Kyungjune Lee, Seongmin Lee

TL;DR

InViTe is a client-server framework that generates high-fidelity, personalized 3D avatars from a single portrait image, complete with realistic hair and customizable makeup. By offloading heavy generative tasks to a server and using Spherical Gaussian approximations for lighting, the system enables real-time, physically-based rendering (PBR) on standard mobile devices.

What Problem Does This Paper Address?

Creating a 3D avatar that actually looks like you—and allows for stylistic customization—usually requires specialized hardware (like 3D scanners), professional artistry, or immense computational power. While 3D Morphable Models (3DMMs) have made 2D-to-3D face reconstruction easier, they often struggle with two things:

  1. Identity Details: Standard models often ignore hair or produce "blurry" textures that don't capture a user's unique look.
  2. Mobile Constraints: High-quality rendering and complex geometry generation are too heavy for mobile processors to handle in real-time without significant lag.

The authors introduce InViTe (Individual Virtual Transfer) to bridge this gap, providing a system that handles reconstruction, personalization (like makeup transfer), and realistic rendering within a mobile-friendly latency envelope.

Methods

The system architecture is split between a Server (for heavy lifting) and a Mobile Client (for tracking and rendering).

  • 3D Mesh Generation: The system uses a 3DMM (specifically the FLAME model) to get the basic face shape. To solve the "hair problem," they employ a strand-based hair estimator that converts 2D hair into a 3D mesh. To keep it mobile-friendly, they consolidate individual strands into a single solid mesh that retains detailed geometry without the overhead of physics-simulated strands.
  • Personalization & Makeup Transfer: A StyleGAN-based architecture is used to "normalize" the face into a high-fidelity UV texture map. Users can provide a "style image" (e.g., a photo of someone wearing specific makeup), and the system uses an alpha-blending map to transfer those makeup attributes onto the user's 3D face while preserving their identity.
  • Mobile Rendering Pipeline: To achieve "immersive" realism, the system uses Physically Based Rendering (PBR). Since mobile devices can't easily process high-resolution environment lighting (HDRI), InViTe approximates lighting using Spherical Gaussians (SG). It optimizes the environment into 19 "lobes" (vectors), allowing for fast, realistic light calculations as the user moves their head.
  • Tracking: The mobile device uses the front-facing camera to track facial landmarks in real-time, deforming the generated 3D mesh to match the user's expressions.

Datasets

  • MT (Makeup Transfer) Dataset: Used to train the makeup transfer network, allowing the model to learn how to extract and apply makeup layers.
  • User Study Data: The authors evaluated the system using 10 volunteers (ages 10–32) and 15 randomly selected makeup images to test the accuracy and subjective appeal of the generated avatars.

Key Insights and Ideas

  • Hybrid Geometry: Instead of choosing between a low-poly "bald" head and a high-poly hair simulation, the authors' method of consolidating strands into a single mesh provides a "middle way" that looks realistic but renders quickly.
  • Lighting as Data Compression: The use of Spherical Gaussians is a clever optimization. By compressing complex 360-degree lighting into a small set of parameters, they achieve the look of high-end PC rendering on a smartphone.
  • Efficiency via Protocol: The paper highlights that how you send data matters. By sending optimized mesh and texture data rather than streaming a video or a raw, uncompressed mesh, they drastically reduce latency, making the interaction feel instantaneous.

Key Results

  • Identity Preservation: Using the ArcFace verification model, the system achieved a 91.4% accuracy in matching the generated 3D avatar back to the original portrait.
  • User Satisfaction: In subjective testing, participants gave the personalization quality a score of 4.3/5.
  • Low Latency: The system achieved a transmission latency of just 0.03 seconds. This is roughly 5.7x faster than traditional video streaming and 10.3x faster than streaming raw 3D meshes, proving it is viable for real-time mobile apps.

Limitations and Future Work

The paper is presented as a "Demonstration Track" entry, focusing on the system's architecture and real-time viability. While it handles makeup and hair well, the authors do not deeply discuss:

  • Extreme Occlusions: How the system handles faces partially covered by hands or glasses during the initial reconstruction.
  • Diverse Hair Textures: While the strand-based estimator is a step up, very curly or complex hairstyles may still pose a challenge for consolidation into a single mesh.
  • Future Directions: The authors suggest this framework could be expanded for more immersive virtual communication and social VR/AR platforms where personalized identity is paramount.

Summary updated

BibTeX
@inproceedings{jang2024invite,
  author = {Mingyu Jang and Kyungjune Lee and Seongmin Lee},
  title = {{InViTe: Individual Virtual Transfer for Personalized 3D Face Generation System}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}

Loading related content...

Figure