F2RPC: Fake to Real Portrait Control From a Virtual Character

Seoungyoon Kang, Minjae Kim, Hyunjung ShimIEEE Access 2023 · DOI

Introduction

Creating convincing "digital clones"—virtual humans that look and move like real people—is a major challenge in computer vision. To overcome the "uncanny valley," a model must excel at two distinct tasks: destylization (transforming a virtual character's synthetic texture into photo-realistic skin) and reenactment (animating the face naturally based on a driving video).

Current state-of-the-art methods typically tackle these tasks separately, often relying on GAN inversion to map images into a StyleGAN latent space. However, GAN inversion is notoriously lossy, struggling to capture the exact identity of the source character and remaining highly sensitive to facial alignment. When destylization and reenactment models are cascaded, these errors compound, leading to "identity drift," visual artifacts, and significant computational overhead.

The F2RPC Framework

F2RPC (Fake to Real Portrait Control) addresses these issues by moving away from GAN inversion in favor of a Blind Face Restoration (BFR) backbone, specifically GPEN. The authors treat the "fake" appearance of a virtual character as a form of image degradation that needs to be "restored" to reality.

The F2RPC process uses a source virtual character (left) and motion from a driving video (right).

The framework consists of two main sub-modules integrated into a single encoder-decoder architecture:

  1. AdaGPEN (Destylization): This module handles the visual appearance. It uses a novel AdaMix block to blend features, exploiting the fact that in StyleGAN-like architectures, low-resolution layers control global structure while high-resolution layers control fine texture. By mixing features from the original image and a blurred version, it corrects unrealistic 3D structures while injecting realistic skin textures.
  2. PCGPEN (Reenactment): This module handles motion through flow-based feature warping at an intermediate resolution. It uses 3D Morphable Model (3DMM) descriptors from a driving video to predict feature movement. Because it operates in the feature domain, the decoder can "fill in" missing information (like the inside of a mouth) using learned facial priors.

By sharing a single encoder between these tasks, F2RPC avoids the information loss that occurs when converting high-dimensional features back into images between stages.

Evaluation and Results

The authors evaluated F2RPC using several key benchmarks, including MSSF for destylization, HDTF for reenactment accuracy, and FFHQ for realism scoring. F2RPC significantly outperformed cascaded baselines (such as DualStyleGAN combined with StyleHEAT):

The final output demonstrates superior photo-realism and identity preservation during movement.
  • Image Quality: Improved the FID score by 26.4%, indicating much higher photo-realism.
  • Identity Preservation: Achieved 95% better identity similarity (CSIM) at 512x512 resolution compared to cascaded baselines.
  • Efficiency: The unified architecture requires 38% fewer FLOPs than StyleHEAT and runs 13.5% faster than a comparable 2-stage version of itself.
  • Robustness: Unlike previous methods that produce artifacts around the neck and hair due to alignment issues, F2RPC remains stable even with unaligned real-world images.

Conclusion

F2RPC demonstrates that Blind Face Restoration is a powerful tool for digital clone creation, offering better identity preservation and efficiency than pure GAN generation. While the model handles standard talking-head motions effectively, it still faces challenges with extreme poses. Future work may explore 3D point-cloud-based feature representations to maintain consistency during complex head rotations and sharp camera angles.

Summary updated

BibTeX
@article{kang2023f,
  author = {Seoungyoon Kang and Minjae Kim and Hyunjung Shim},
  title = {{F2RPC: Fake to Real Portrait Control From a Virtual Character}},
  journal = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3306836},
}
Figure