InViTe: Individual Virtual Transfer for Personalized 3D Face Generation System

Mingyu Jang, Kyungjune Lee, Seongmin Lee, Hoseok Tong, Juwan Chung, Yusung Ro, Sanghoon Lee

Introduction

Creating personalized, high-fidelity 3D avatars on mobile devices has traditionally been a trade-off between visual realism and computational performance. While high-quality 3D face reconstruction typically requires specialized hardware or multi-view camera setups, mobile applications often struggle with the heavy computational load of Physically Based Rendering (PBR) and complex geometry like hair. The InViTe (Individual Virtual Transfer) system addresses these challenges through a novel client-server architecture that offloads intensive generation tasks while maintaining real-time interactivity on standard smartphones.

Overview of the InViTe system: (a) User setup with a ring light and smartphone, (b) Generated 3D assets including mesh, UV texture, and lighting maps, (c) Final rendered results with personalized makeup transfer.

System Architecture and Methodology

The InViTe framework is divided into three primary pipelines designed to balance server-side generation with client-side rendering efficiency.

1. 3D Mesh Generation

The system utilizes the FLAME (3D Morphable Model) to estimate facial geometry from a single 2D portrait. A significant innovation is the "Hybrid Hair Approach." To avoid the high rendering cost of individual hair strands on mobile GPUs, the system first estimates strand-based hair geometry and then consolidates these directions into a single, solid 3D mesh. This "baked" mesh preserves the visual complexity of the hairstyle while remaining lightweight enough for mobile rendering.

2. Personalization and Makeup Transfer

To ensure the avatar maintains the user's identity, the system employs a StyleGAN-based face normalization technique. This process generates a high-quality UV texture map by filling in occluded areas and correcting for environmental lighting. Furthermore, users can apply specific makeup styles via a generative model that creates an alpha blending map. This allows the system to overlay makeup as an additive layer, preserving the underlying facial features and preventing the "hallucination" of a different identity.

The core assets generated by the InViTe server: 3D face and hair meshes, a normalized face UV texture, and an environment map with Spherical Gaussian lighting parameters.

3. Mobile Rendering with Spherical Gaussians

To achieve realistic lighting without the bandwidth overhead of high-resolution HDR images, InViTe uses Spherical Gaussian (SG) parameters. By approximating complex environment lighting into a compact set of 19 lobes, the system reduces the data transmitted to the mobile client. The mobile device then uses these parameters for fast, high-quality PBR rendering.

Evaluation and Results

The system was evaluated based on latency, identity preservation, and user satisfaction.

  • Performance: InViTe achieved an ultra-low data transmission latency of 0.03 seconds. This performance is approximately 5.7 times faster than video streaming and 10.3 times faster than raw mesh streaming, enabling a seamless user experience.
  • Fidelity: Using the ArcFace benchmark for face verification, the generated 3D models achieved a 91.4% accuracy rate, demonstrating high recognizable identity preservation.
  • User Experience: In subjective evaluations, the personalization features (makeup and hair) received an average score of 4.3 out of 5, indicating that the generated avatars felt natural and accurate to the users.

Conclusion

InViTe demonstrates that high-fidelity 3D avatar generation is feasible on mobile devices by strategically splitting tasks between a server and a client. By treating lighting as compact data (SG parameters) and using a hybrid approach for complex geometry like hair, the system provides a realistic and personalized experience with minimal latency. Future work may involve extending the system to support dynamic facial expressions and improving texture extraction under extreme occlusions.

Summary updated

BibTeX
@inproceedings{jang2024invite,
  author = {Mingyu Jang and Kyungjune Lee and Seongmin Lee and Hoseok Tong and Juwan Chung and Yusung Ro and Sanghoon Lee},
  title = {{InViTe: Individual Virtual Transfer for Personalized 3D Face Generation System}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure