PixSwap: High-Resolution Face Swapping for Effective Reflection of Identity via Pixel-Level Supervision with Synthetic Paired Dataset

Taewoo Kim, Geonsu Lee, Hyukgi Lee, Seongtae Kim, Younggun LeeWACV 2025

TL;DR

This paper introduces PixSwap, a novel face-swapping method that achieves state-of-the-art high-resolution results by addressing two key limitations: the lack of pixel-level supervision and suboptimal architectures in existing StyleGAN-based approaches. By constructing a synthetic paired dataset and proposing a modified StyleGAN generator, PixSwap significantly improves identity reflection while preserving target attributes, even in challenging cross-identity scenarios.

What Problem Does This Paper Address?

Face swapping, the task of exchanging facial identities between images or videos while preserving attributes like expression, pose, and skin color, has seen remarkable progress. However, generating high-resolution swapped faces that accurately reflect the source identity and maintain target attributes, especially in cross-identity scenarios (where source and target have different identities), remains a challenge. Existing methods, particularly those leveraging StyleGAN for high-resolution synthesis, often struggle with robust identity reflection. This is attributed to two main issues:

  1. Lack of Pixel-Level Supervision: Most methods lack direct pixel-level guidance on how to accurately blend source identity with target attributes, leading to artifacts or incomplete identity transfer.
  2. Suboptimal Architectures and Pipelines: StyleGAN-based methods can underutilize source latent vectors or rely heavily on pre-trained models, leading to insufficient identity representation or reduced robustness due to cascading errors.

Methods

PixSwap tackles these challenges with two primary innovations:

  1. Synthetic Paired Dataset Construction: To provide crucial pixel-level supervision, the authors construct a pseudo-ground truth paired dataset. This involves:
  2. Generating initial face swaps using a pre-trained model.
  3. Selecting high-quality swaps that preserve source identity (verified by a face recognition model).
  4. Transferring the skin color from the target image to the swapped image to reduce discrepancies.
  5. Employing a super-resolution network to enhance the quality of the facial region. This synthetic dataset allows for effective pixel-level supervision, guiding the model to better reflect source identity and maintain target attributes.

  6. Modified StyleGAN Generator: The authors propose a novel generator architecture that leverages StyleGAN's strengths while addressing its limitations for face swapping. Key modifications include:

  7. Enhanced Spatial Information Utilization: The generator is designed to better utilize the spatial features (F) of the target attributes and the latent vectors (W+) from the source identity.
  8. Increased Dilation Factor: By increasing the dilation factor in early convolutional layers of the StyleGAN blocks, the generator can process features from a higher resolution (e.g., 32x32 instead of 4x4), allowing for better incorporation of spatial information from the target attributes.
  9. Complementary Attribute Encoding: An attribute encoder maps target attributes to W+ space, complementing the spatial features and ensuring global attribute consistency.

The training objective comprises three losses: reconstruction loss (L2 + LPIPS) for pixel-level accuracy, identity loss (using ArcFace embeddings) to ensure source identity reflection, and an adversarial loss for realistic image generation.

Datasets

  • Training: The model is trained on the FFHQ dataset. A synthetic paired dataset of 300,000 samples (source image, target image, pseudo ground truth) is generated for pixel-level supervision.
  • Evaluation:
    • Quantitative evaluation is performed on 10,000 randomly sampled pairs from the CelebA-HQ dataset.
  • Qualitative evaluation uses CelebA-HQ for image results and the CelebV-HQ video dataset for video-level testing.

Key Insights and Ideas

  • Pixel-level supervision is critical for robust cross-identity face swapping: The synthetic paired dataset significantly boosts performance, demonstrating that explicit guidance on how to blend identities and attributes is more effective than relying solely on implicit learning.
  • StyleGAN's latent space and feature maps can be better leveraged: By modifying the generator's architecture to process higher-resolution spatial features and inject source identity information more effectively, PixSwap achieves superior identity reflection.
  • Balancing identity and attributes is key: The proposed architecture and training strategy effectively balance preserving target attributes (like expression, pose) with accurately reflecting the source identity, avoiding common pitfalls like attribute leakage or identity loss.
  • Simplicity can be powerful: PixSwap achieves state-of-the-art results using only three loss functions and without relying on multiple pre-trained networks during inference, making it more efficient and robust.

Key Results

  • Quantitative: PixSwap achieves state-of-the-art performance across multiple metrics, including identity similarity, shape distance, expression distance, pose distance, and image quality score (IQS). Notably, it shows a significant improvement in identity similarity compared to all baselines, especially when trained with the paired dataset.
  • Qualitative: The results demonstrate PixSwap's ability to generate sharp, realistic high-resolution images and videos with highly accurate source identity reflection and well-preserved target attributes. Visual comparisons show a clear advantage over existing StyleGAN-based and non-StyleGAN-based methods, particularly in maintaining identity consistency across video frames and reducing flickering.

Limitations and Future Work

  • Dataset Limitation: The model is trained solely on the FFHQ dataset. The authors suggest that training on larger-scale face recognition datasets or video datasets could further improve performance.
  • Extreme Pose and Eye Gaze: The model may struggle with extreme pose cases and occasionally exhibits artificial appearances or inaccurate eye gazing.
  • Future Directions: The authors propose exploring larger datasets and improving the pseudo-ground truth generation process to better capture target eye-gaze. Incorporating methods for generating more realistic images is also suggested.

Summary updated

BibTeX
@inproceedings{kim2025pixswap,
  author = {Taewoo Kim and Geonsu Lee and Hyukgi Lee and Seongtae Kim and Younggun Lee},
  title = {{PixSwap: High-Resolution Face Swapping for Effective Reflection of Identity via Pixel-Level Supervision with Synthetic Paired Dataset}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure