Abstract
Recent advances in artificial intelligence technology have significantly improved facial image manipulation, known as Deepfake. This study focuses on "expression swap," which manipulates facial expressions in real-time without replacing the entire face. We propose an evaluation framework for expression swap models in online class environments across three scenarios: attendance check, presentation, and examination. Evaluating the First Order Motion Model (FOMM) and GANimation, we find that while both perform well in close-ups, their quality degrades significantly as the face occupies a smaller portion of the frame. Finally, we demonstrate the feasibility of these models for real-time use on platforms like Zoom and Google Meet.
Introduction
The rapid development of deep learning has led to sophisticated facial image manipulation techniques. These are generally categorized into four types: entire face synthesis, identity swap, attribute manipulation, and expression swap.
While identity swapping is popular, this research focuses on expression swap (face reenactment). This technique is particularly advantageous for real-time applications because it only modifies expressions, preserving the original identity and background, which requires less computational overhead than full-face synthesis. In the context of online education, this presents a unique security threat where students could appear attentive using a static photo and a proxy's movements.
Methodology
The study evaluates two prominent models:
- First Order Motion Model (FOMM): A self-supervised approach that uses learned keypoints and affine transformations to warp a source image based on target motion.
- GANimation: A GAN-based approach utilizing Action Units (AUs) to manipulate specific facial muscles via attention masks.
To simulate real-world online classes, the researchers defined three scenarios based on the "face-to-frame" ratio:
- Scenario 1 (Attendance): Close-up, face >50% of the frame.
- Scenario 2 (Examination): Mid-shot, face ~10% of the frame.
- Scenario 3 (Presentation): Wide-shot, face ~3% of the frame.
Key Results and Discussion
The Impact of Face Scale
A critical finding is the performance degradation as the subject moves further from the camera. Models trained on cropped facial datasets struggle with localization in wide-angle shots. In Scenario 1, FOMM achieved a high Universal Quality Image Index (UQI) of 0.96. However, in Scenario 3, both models produced significant artifacts, with GANimation often rendering the face unrecognizable.
Model Trade-offs
The evaluation revealed distinct characteristics for each model. FOMM maintains better structural integrity and handles head rotations more smoothly, making it superior for general video quality. Conversely, GANimation is more adept at capturing subtle, complex muscle movements but is prone to "image damage" if the input alignment is suboptimal.
Real-time Feasibility
Latency is the primary hurdle for live deception. The framework achieved processing times of 0.3s for FOMM and 0.5s for GANimation. When integrated with virtual camera software, these manipulated streams were successfully broadcast on Zoom and Microsoft Teams, appearing as minor network lag to the end-user.
Conclusion
This study highlights the vulnerabilities of online classroom environments to real-time Deepfake manipulation. While current models struggle with small-scale facial features in wide-angle scenarios, they are highly effective in close-up settings. Future work will focus on improving temporal consistency to eliminate frame-to-frame flickering and enhancing robustness for varying environmental conditions.
Summary updated