Introduction
Face swapping—the process of transferring a source person's identity onto a target video while preserving expressions and backgrounds—has traditionally been a computationally intensive task. Leading models like FaceShifter and SimSwap require hundreds of millions of parameters and tens of billions of floating-point operations (FLOPs) per frame. This heavy computational load makes real-time execution on mobile devices nearly impossible. Furthermore, standard compression techniques often fail in this domain, as shrinking the network typically leads to "identity leakage" or significant visual artifacts.
MobileFaceSwap addresses these challenges by introducing a hyper-efficient framework designed for high-quality, real-time video face swapping on edge devices. By utilizing a dynamic neural network that adapts its parameters based on the source identity, the authors achieved a 100x reduction in computational costs compared to state-of-the-art models while maintaining comparable visual fidelity.
Methods: Identity-aware Dynamic Networks
The core innovation of MobileFaceSwap is the Identity-aware Dynamic Network (IDN). Unlike static models with fixed weights, the IDN dynamically adjusts its filters based on the specific source face being processed. This allows a small network to specialize itself for a specific identity during inference, maximizing efficiency.
Key Technical Components
- Identity Injection Module (IIM): This module serves as the heart of the system. It employs two dynamic techniques: Weights Prediction for depthwise convolutions and Weights Modulation for pointwise convolutions. The source face's identity embedding (extracted via a pre-trained ArcFace model) acts as a set of instructions to "re-wire" the IDN's filters on the fly.
- Knowledge Distillation (KD) with Loss Reweighting: To train the lightweight "student" model, the authors used much larger "teacher" models (SimSwap and FaceShifter) as references. Recognizing that teachers can sometimes produce artifacts, they designed a Loss Reweighting Module to evaluate the quality of the teacher's output. If the teacher produces a poor swap, the student's loss is down-weighted to prevent it from learning errors.
- Weakly Semantic Fusion Module: To ensure temporal stability and seamless blending, this lightweight module predicts a fusion mask that integrates the swapped face into the original target frame, preserving hair and background details.
Datasets and Training
The model was trained on a refined subset of VGGFace2, consisting of 550,000 images selected for high-quality facial features. Quantitative benchmarks were conducted on the FaceForensics++ dataset, while qualitative testing utilized CelebA-HQ and high-resolution web images to demonstrate real-world performance.
Key Results and Performance
The efficiency gains of MobileFaceSwap are significant. The IDN utilizes only 0.50M parameters and 0.33G FLOPs, representing a 146x reduction in computation compared to SimSwap (107M parameters and 55.7G FLOPs). On a MediaTek Dimensity 1100 mobile chip, the model achieves 26 FPS without requiring hardware-specific quantization.
In terms of visual fidelity, the model achieved an identity retrieval accuracy of 95.98%, which is comparable to its much larger teacher models. Qualitative results indicate that MobileFaceSwap often produces fewer artifacts around the eyes and forehead than SimSwap, likely due to the quality-aware distillation process.
Conclusion
MobileFaceSwap demonstrates that dynamic networks can bridge the gap between high-end facial manipulation and mobile hardware constraints. By treating identity information as a modulator for weights rather than just a feature map, the framework achieves deep identity integration with minimal parameters. While the authors acknowledge the potential for misuse in creating malicious forgeries, they hope this efficient framework will also aid in the development of more robust deepfake detection tools.