Real-time face swapping as a tool for understanding infant self-recognition

Sao Mai Nguyen, Masaki Ogino, Minoru AsadaarXiv 2011 · arXiv

Introduction

The development of self-recognition is a fundamental milestone in human psychology. Researchers often distinguish between contingency—the immediate temporal link between one's actions and visual feedback—and familiarity—the recognition of one's own facial features. While infants detect movement contingency shortly after birth, they typically do not recognize their own features until approximately 18 months of age.

To investigate this transition, psychologists require tools that can "decouple" identity from motion. For instance, showing an infant a stranger's face that moves in perfect synchrony with their own head. Previous attempts at this were hindered by high latency or the need for invasive markers. This paper presents a markerless, high-speed face-swapping system designed to bridge this gap.

System Architecture

The proposed system operates in real-time by combining a 3D visual tracker with a pose-dependent face swapper.

Overview of the real-time face-swapping pipeline, showing head posture estimation and the selection of replacement faces.

The pipeline consists of three primary stages:

  1. 3D Visual Tracking: Instead of complex mesh modeling, the system utilizes a generic 3D ellipsoid model. A particle filter based on sparse template matching estimates a 10-dimensional state vector, capturing the subject's head position and orientation.
  2. Face Swapping: A library of "replacement faces" (Subject B) is pre-captured across various orientations. The system selects the image closest to the estimated pose of Subject A and applies temporal interpolation for smooth transitions.
  3. GPGPU Acceleration: To maintain the illusion of contingency, the system must operate with minimal delay. By parallelizing tracking and rendering tasks using NVIDIA CUDA, the researchers achieved a total system latency of just 99ms.

Performance and Robustness

The system was validated against a professional motion capture system, yielding an average orientation error of only 9°. Beyond accuracy, the system demonstrates significant robustness in environments typical of developmental studies.

Example of the face-swapping system operating in a cluttered environment with multiple subjects.

Key performance highlights include:

  • Occlusion Handling: The ellipsoid model remains stable even when infants touch their faces or hold toys, which frequently causes traditional trackers to fail.
  • Wide Range of Motion: The tracker maintains lock at pitch angles up to 70°.
  • Environmental Stability: The system effectively ignores background distractors, such as other people moving in the camera's field of view.

Conclusion and Future Directions

By providing a non-invasive way to manipulate visual feedback, this tool opens new avenues for studying infant development and conditions like Autistic Spectrum Disorder (ASD). Future work aims to incorporate dynamic facial expression mapping, allowing the system to transfer smiles or frowns from the source to the target face. Additionally, the framework may inform the development of "self-consciousness" in robotic systems by helping AI correlate motor commands with visual outcomes.

Summary updated

Abstract

To study the preference of infants for contingency of movements and familiarity of faces during self-recognition task, we built, as an accurate and instantaneous imitator, a real-time face- swapper for videos. We present a non-constraint face-swapper based on 3D visual tracking that achieves real-time performance through parallel computing. Our imitator system is par- ticularly suited for experiments involving children with Autistic Spectrum Disorder who are often strongly disturbed by the constraints of other methods.
BibTeX
@misc{nguyen2011real,
  author = {Sao Mai Nguyen and Masaki Ogino and Minoru Asada},
  title = {{Real-time face swapping as a tool for understanding infant self-recognition}},
  year = {2011},
  eprint = {1112.2095},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1112.2095},
}
Figure