Show Your Face: Restoring Complete Facial Images from Partial Observations for VR Meeting

Zheng Chen, Zhiqi Zhang, Junsong Yuan, Yi Xu, Lantao Liu

Abstract

Visual contact is a cornerstone of immersive social interaction, yet Virtual Reality (VR) headsets physically block a significant portion of a user's face. This paper presents FIC-Net, a practical, lightweight 2D framework that reconstructs full, expressive facial images from partial infrared (IR) views captured by internal headset cameras. By separating the problem into local geometry correction and global color fusion, the proposed method achieves high-fidelity results that outperform complex 3D-based approaches and standard GANs, all while maintaining real-time performance on mobile hardware.

Introduction

In the context of VR meetings, the "immersion killer" is often the headset itself. While users can see each other's avatars, the physical device obscures the eyes and upper face, preventing the subtle non-verbal cues essential for presence and comfort.

Comparison between original VR views with headsets (left) and the restored facial images (right).

Existing solutions typically rely on person-specific 3D models, which are computationally intensive and difficult to scale. This work introduces a 2D alternative that leverages partial observations to restore a complete, photorealistic face.

The Challenge of Facial Occlusion

Headset Mounted Cameras (HMCs) face three primary hurdles:

  1. Partial Observation: Cameras only capture "patches" of the face (e.g., eyes and mouth) because the headset occupies the rest of the field of view.
  2. Extreme Distortion: Positioned at steep angles very close to the face, HMCs produce massive non-linear perspective distortions.
  3. Lack of Color: To operate in the dark environment inside a headset, HMCs typically use infrared sensors, resulting in grayscale data.
Diverse users in a VR meeting environment experiencing varying degrees of facial occlusion.

FIC-Net Architecture

The authors propose the Facial Image Composition Network (FIC-Net), which employs a "local-to-global" hierarchy to address these challenges.

Local Transformation Models

Three independent autoencoders—one for each eye and one for the mouth—process the tilted, grayscale IR patches. These models focus exclusively on geometry, "frontalizing" the distorted side-views into a standard straight-on perspective. A key finding was that standard skip connections (common in U-Nets) were detrimental here, as they passed along incorrect spatial information from the distorted inputs.

Global Fusion Model

The frontalized patches are then passed to a GAN-based global fusion model. This stage merges the patches with a pre-recorded RGB reference image of the user. By using the reference image as a prior, the model ensures the reconstructed patches match the user's skin tone and the surrounding lighting, avoiding the need to "hallucinate" color from IR data.

The FIC Dataset

To train and evaluate the model, the researchers introduced the Facial Image Composition (FIC) Dataset. Derived from the MEAD (Multi-view Emotional Audio-visual Dataset), it includes over 36,000 images across 14 identities and 8 expressions. The dataset simulates HMC views by cropping specific angles and converting them to grayscale, providing a robust benchmark for both seen and unseen identities.

Results and Performance

FIC-Net demonstrates significant improvements over baselines such as StyleMapGAN and Poisson Blending. Quantitatively, it leads in metrics like SSIM (Structural Similarity), PSNR (Peak Signal-to-Noise Ratio), and LPIPS (Perceptual Similarity). Qualitatively, the model preserves fine details—such as teeth and eyelids—that other methods often blur.

Crucially, FIC-Net maintains a runtime of approximately 0.01 seconds per frame, making it highly viable for real-time VR communication on mobile hardware.

Conclusion and Future Work

FIC-Net provides a scalable, efficient solution for restoring facial visibility in VR. By decoupling geometry and color, it achieves high fidelity without the overhead of 3D modeling. Future research will focus on improving skin tone diversity in reconstruction and ensuring temporal consistency to prevent flickering in video streams.

Reference: Chen, Z., et al. "Show Your Face: Restoring Complete Facial Images from Partial Observations for VR Meeting." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2023.

Summary updated

Abstract

Virtual Reality (VR) headsets allow users to interact with the virtual world. However, the device physically blocks visual connections among users, causing huge inconveniences for VR meetings. To address this issue, studies have been conducted to restore human faces from images captured by Headset Mounted Cameras (HMC). Unfortunately, existing approaches heavily rely on high-resolution person-specific 3D models which are prohibitively expensive to apply to large-scale scenarios. Our goal is to design an efficient framework for restoring users’ facial data in VR meetings. Specifically, we first build a new dataset, named Facial Image Composition (FIC) data which approximates the real HMC images from a VR headset. By leveraging the heterogeneity of the HMC images, we decompose the restoration problem into a local geometry transformation and global color/style fusion. Then we propose a 2D light-weight facial image composition network (FIC-Net), where three independent local models are responsible for transforming raw HMC patches and the global model performs a fusion of the transformed HMC patches with a pre-recorded reference image. Finally, we also propose a stage-wise training strategy to optimize the generalization of our FIC-Net. We have validated the effectiveness of our proposed FIC-Net through extensive experiments.
BibTeX
@inproceedings{chen2024show,
  author = {Zheng Chen and Zhiqi Zhang and Junsong Yuan and Yi Xu and Lantao Liu},
  title = {{Show Your Face: Restoring Complete Facial Images from Partial Observations for VR Meeting}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00849},
}
Figure