Abstract
Visual contact is a cornerstone of immersive social interaction, yet Virtual Reality (VR) headsets physically block a significant portion of a user's face. This paper presents FIC-Net, a practical, lightweight 2D framework that reconstructs full, expressive facial images from partial infrared (IR) views captured by internal headset cameras. By separating the problem into local geometry correction and global color fusion, the proposed method achieves high-fidelity results that outperform complex 3D-based approaches and standard GANs, all while maintaining real-time performance on mobile hardware.
Introduction
In the context of VR meetings, the "immersion killer" is often the headset itself. While users can see each other's avatars, the physical device obscures the eyes and upper face, preventing the subtle non-verbal cues essential for presence and comfort.
Existing solutions typically rely on person-specific 3D models, which are computationally intensive and difficult to scale. This work introduces a 2D alternative that leverages partial observations to restore a complete, photorealistic face.
The Challenge of Facial Occlusion
Headset Mounted Cameras (HMCs) face three primary hurdles:
- Partial Observation: Cameras only capture "patches" of the face (e.g., eyes and mouth) because the headset occupies the rest of the field of view.
- Extreme Distortion: Positioned at steep angles very close to the face, HMCs produce massive non-linear perspective distortions.
- Lack of Color: To operate in the dark environment inside a headset, HMCs typically use infrared sensors, resulting in grayscale data.
FIC-Net Architecture
The authors propose the Facial Image Composition Network (FIC-Net), which employs a "local-to-global" hierarchy to address these challenges.
Local Transformation Models
Three independent autoencoders—one for each eye and one for the mouth—process the tilted, grayscale IR patches. These models focus exclusively on geometry, "frontalizing" the distorted side-views into a standard straight-on perspective. A key finding was that standard skip connections (common in U-Nets) were detrimental here, as they passed along incorrect spatial information from the distorted inputs.
Global Fusion Model
The frontalized patches are then passed to a GAN-based global fusion model. This stage merges the patches with a pre-recorded RGB reference image of the user. By using the reference image as a prior, the model ensures the reconstructed patches match the user's skin tone and the surrounding lighting, avoiding the need to "hallucinate" color from IR data.
The FIC Dataset
To train and evaluate the model, the researchers introduced the Facial Image Composition (FIC) Dataset. Derived from the MEAD (Multi-view Emotional Audio-visual Dataset), it includes over 36,000 images across 14 identities and 8 expressions. The dataset simulates HMC views by cropping specific angles and converting them to grayscale, providing a robust benchmark for both seen and unseen identities.
Results and Performance
FIC-Net demonstrates significant improvements over baselines such as StyleMapGAN and Poisson Blending. Quantitatively, it leads in metrics like SSIM (Structural Similarity), PSNR (Peak Signal-to-Noise Ratio), and LPIPS (Perceptual Similarity). Qualitatively, the model preserves fine details—such as teeth and eyelids—that other methods often blur.
Crucially, FIC-Net maintains a runtime of approximately 0.01 seconds per frame, making it highly viable for real-time VR communication on mobile hardware.
Conclusion and Future Work
FIC-Net provides a scalable, efficient solution for restoring facial visibility in VR. By decoupling geometry and color, it achieves high fidelity without the overhead of 3D modeling. Future research will focus on improving skin tone diversity in reconstruction and ensuring temporal consistency to prevent flickering in video streams.
Reference: Chen, Z., et al. "Show Your Face: Restoring Complete Facial Images from Partial Observations for VR Meeting." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.
Summary updated