An experimental study of animating-based facial image manipulation in online class environments

Jeong-Ha Park, Chae-Yun Lim, Hyuk-Yoon KwonScientific 2023 · DOI

Abstract

Recent advances in artificial intelligence technology have significantly improved facial image manipulation, known as Deepfake. This study focuses on "expression swap," which manipulates facial expressions in real-time without replacing the entire face. We propose an evaluation framework for expression swap models in online class environments across three scenarios: attendance check, presentation, and examination. Evaluating the First Order Motion Model (FOMM) and GANimation, we find that while both perform well in close-ups, their quality degrades significantly as the face occupies a smaller portion of the frame. Finally, we demonstrate the feasibility of these models for real-time use on platforms like Zoom and Google Meet.

Introduction

The rapid development of deep learning has led to sophisticated facial image manipulation techniques. These are generally categorized into four types: entire face synthesis, identity swap, attribute manipulation, and expression swap.

Identity swap involves replacing the target's identity with a source face.

While identity swapping is popular, this research focuses on expression swap (face reenactment). This technique is particularly advantageous for real-time applications because it only modifies expressions, preserving the original identity and background, which requires less computational overhead than full-face synthesis. In the context of online education, this presents a unique security threat where students could appear attentive using a static photo and a proxy's movements.

Methodology

The study evaluates two prominent models:

  1. First Order Motion Model (FOMM): A self-supervised approach that uses learned keypoints and affine transformations to warp a source image based on target motion.
  2. GANimation: A GAN-based approach utilizing Action Units (AUs) to manipulate specific facial muscles via attention masks.
Expression swap animates a source image using the movements of a target video.

To simulate real-world online classes, the researchers defined three scenarios based on the "face-to-frame" ratio:

  • Scenario 1 (Attendance): Close-up, face >50% of the frame.
  • Scenario 2 (Examination): Mid-shot, face ~10% of the frame.
  • Scenario 3 (Presentation): Wide-shot, face ~3% of the frame.

Key Results and Discussion

The Impact of Face Scale

A critical finding is the performance degradation as the subject moves further from the camera. Models trained on cropped facial datasets struggle with localization in wide-angle shots. In Scenario 1, FOMM achieved a high Universal Quality Image Index (UQI) of 0.96. However, in Scenario 3, both models produced significant artifacts, with GANimation often rendering the face unrecognizable.

Model Trade-offs

The evaluation revealed distinct characteristics for each model. FOMM maintains better structural integrity and handles head rotations more smoothly, making it superior for general video quality. Conversely, GANimation is more adept at capturing subtle, complex muscle movements but is prone to "image damage" if the input alignment is suboptimal.

Real-time Feasibility

Latency is the primary hurdle for live deception. The framework achieved processing times of 0.3s for FOMM and 0.5s for GANimation. When integrated with virtual camera software, these manipulated streams were successfully broadcast on Zoom and Microsoft Teams, appearing as minor network lag to the end-user.

Conclusion

This study highlights the vulnerabilities of online classroom environments to real-time Deepfake manipulation. While current models struggle with small-scale facial features in wide-angle scenarios, they are highly effective in close-up settings. Future work will focus on improving temporal consistency to eliminate frame-to-frame flickering and enhancing robustness for varying environmental conditions.

Summary updated

Abstract

Recent advances in artificial intelligence technology have significantly improved facial image manipulation, which is known as Deepfake. Facial image manipulation synthesizes or replaces a region of the face in an image with that of another face. The techniques for facial image manipulation are classified into four categories: (1) entire face synthesis, (2) identity swap, (3) attribute manipulation, and (4) expression swap. Out of them, we focus on expression swap because it effectively manipulates only the expression of the face in the images or videos without creating or replacing the entire face, having advantages for the real-time application. In this study, we propose an evaluation framework of the expression swap models targeting the real-time online class environments. For this, we define three kinds of scenarios according to the portion of the face in the entire image considering actual online class situations: (1) attendance check (Scenario 1), (2) presentation (Scenario 2), and (3) examination (Scenario 3). Considering the manipulation on the online class environments, the framework receives a single source image and a target video and generates the video that manipulates a face of the target video to that in the source image. To this end, we select two models that satisfy the conditions required by the framework: (1) first order model and (2) GANimation. We implement these models in the framework and evaluate their performance for the defined scenarios. Through the quantitative and qualitative evaluation, we observe distinguishing properties of the used two models. Specifically, both models show acceptable results in Scenario 1, where the face occupies a large portion of the image. However, their performances are significantly degraded in Scenarios 2 and 3, where the face occupies less portion of the image; the first order model causes relatively less loss of image quality than GANimation in the result of the quantitative evaluation. In contrast, GANimation has the advantages of representing facial expression changes compared to the first order model. Finally, we devise an architecture for applying the expression swap model to the online video conferencing application in real-time. In particular, by applying the expression swap model to widely used online meeting platforms such as Zoom, Google Meet, and Microsoft Teams, we demonstrate its feasibility for real-time online classes.
BibTeX
@inproceedings{park2023experimental,
  author = {Jeong-Ha Park and Chae-Yun Lim and Hyuk-Yoon Kwon},
  title = {{An experimental study of animating-based facial image manipulation in online class environments}},
  booktitle = {Scientific},
  year = {2023},
  doi = {10.1038/s41598-023-31408-y},
}
Figure