Introduction
Creating personalized, high-fidelity 3D avatars on mobile devices has traditionally been a trade-off between visual realism and computational performance. While high-quality 3D face reconstruction typically requires specialized hardware or multi-view camera setups, mobile applications often struggle with the heavy computational load of Physically Based Rendering (PBR) and complex geometry like hair. The InViTe (Individual Virtual Transfer) system addresses these challenges through a novel client-server architecture that offloads intensive generation tasks while maintaining real-time interactivity on standard smartphones.
System Architecture and Methodology
The InViTe framework is divided into three primary pipelines designed to balance server-side generation with client-side rendering efficiency.
1. 3D Mesh Generation
The system utilizes the FLAME (3D Morphable Model) to estimate facial geometry from a single 2D portrait. A significant innovation is the "Hybrid Hair Approach." To avoid the high rendering cost of individual hair strands on mobile GPUs, the system first estimates strand-based hair geometry and then consolidates these directions into a single, solid 3D mesh. This "baked" mesh preserves the visual complexity of the hairstyle while remaining lightweight enough for mobile rendering.
2. Personalization and Makeup Transfer
To ensure the avatar maintains the user's identity, the system employs a StyleGAN-based face normalization technique. This process generates a high-quality UV texture map by filling in occluded areas and correcting for environmental lighting. Furthermore, users can apply specific makeup styles via a generative model that creates an alpha blending map. This allows the system to overlay makeup as an additive layer, preserving the underlying facial features and preventing the "hallucination" of a different identity.
3. Mobile Rendering with Spherical Gaussians
To achieve realistic lighting without the bandwidth overhead of high-resolution HDR images, InViTe uses Spherical Gaussian (SG) parameters. By approximating complex environment lighting into a compact set of 19 lobes, the system reduces the data transmitted to the mobile client. The mobile device then uses these parameters for fast, high-quality PBR rendering.
Evaluation and Results
The system was evaluated based on latency, identity preservation, and user satisfaction.
- Performance: InViTe achieved an ultra-low data transmission latency of 0.03 seconds. This performance is approximately 5.7 times faster than video streaming and 10.3 times faster than raw mesh streaming, enabling a seamless user experience.
- Fidelity: Using the ArcFace benchmark for face verification, the generated 3D models achieved a 91.4% accuracy rate, demonstrating high recognizable identity preservation.
- User Experience: In subjective evaluations, the personalization features (makeup and hair) received an average score of 4.3 out of 5, indicating that the generated avatars felt natural and accurate to the users.
Conclusion
InViTe demonstrates that high-fidelity 3D avatar generation is feasible on mobile devices by strategically splitting tasks between a server and a client. By treating lighting as compact data (SG parameters) and using a hybrid approach for complex geometry like hair, the system provides a realistic and personalized experience with minimal latency. Future work may involve extending the system to support dynamic facial expressions and improving texture extraction under extreme occlusions.
Summary updated