Abstract
In this paper, a framework of video face replacement is proposed and it deals with the flicker of swapped face in video sequence. This framework contains two main innovations: 1) the technique of image registration is exploited to align the source and target video faces for eliminating the flicker or jitter of the segmented video face sequence; 2) a fast subpixel image registration method is proposed for farther accuracy and efficiency. Unlike the priori works, it minimizes the overlapping region and takes spatiotemporal coherence into account. Flicker in resulted videos is usually caused by the frequently changed bound of the blending target face and unregistered faces between and along video sequences. The subpixel image registration method is proposed to solve the flicker problem. During the alignment process, integer pixel registration is formulated by maximizing the similarity of images with down sampling strategy speeding up the process and sub-pixel image registration is a single-step image match via analytic method. Experimental results show the proposed algorithm reduces the computation time and gets a high accuracy when conducting experiments on different data sets.
Introduction
Video face replacement (or "face swapping") has become a popular tool in film production and social media. However, a common artifact is flicker: the swapped face appears to "jitter" or vibrate slightly from frame to frame. This phenomenon typically arises from two main sources: misalignment between the source face and the target face's position/orientation, and temporal incoherence where small errors in tracking or landmark detection cause blending boundaries to shift inconsistently.
While 3D morphable models can mitigate these issues, they are computationally expensive and require extensive 3D scan libraries. This paper proposes a robust 2D registration-based framework that eliminates visual flicker by precisely aligning faces using a novel, high-speed analytic sub-pixel registration method.
Proposed Framework
The proposed pipeline consists of several integrated stages designed for both accuracy and speed.
- Detection & Tracking: Initial face detection is performed using the Viola-Jones algorithm, followed by the Struck algorithm for robust tracking across frames. Facial landmarks are then located using the Supervised Descent Method (SDM).
- Coarse Registration: To establish a general position, the system employs an image pyramid and downsampling strategy. This "coarse" search identifies the best integer-pixel match by maximizing an Alignment Metric (AM) based on intensity variance.
- Analytic Sub-pixel Registration: This is the core innovation. Unlike traditional methods that iteratively search through interpolated high-resolution images, this framework treats alignment as a continuous mathematical function. By applying bilinear interpolation and calculating partial derivatives of the cross-variance between frames, the authors derive a fifth-order polynomial equation. Solving this equation provides the exact sub-pixel shift ($x, y$) in a single step, bypassing expensive iterative checks.
- Blending: Finally, Poisson Image Editing is used to seamlessly integrate the registered source face into the target frame, ensuring color and lighting consistency.
Key Insights
The researchers treat flicker primarily as a 2D registration failure. By maximizing the overlapping region between adjacent frames and achieving sub-pixel registration accuracy, temporal jitter is naturally minimized. The shift from an iterative search to an analytic solution allows the system to calculate optimal alignment mathematically, significantly boosting efficiency. Furthermore, a sparse matrix approach during coarse matching reduces unnecessary calculations in "flat" image regions.
Experimental Results
The framework was evaluated using the Max Planck Institute Video Face Database for static head scenarios and standard video sequences for dynamic real-world conditions.
Key performance metrics include:
- Reduced Jitter: The proposed method achieved a jitter variance of 1.02–1.15, a marked improvement over existing methods like Bitouk et al. (1.73–2.16) and Afifi et al. (2.28–7.95).
- Computational Efficiency: For a 1000x1000 image, the method required only 0.956 seconds, nearly four times faster than comparable sub-pixel techniques (e.g., Yu et al. at 3.776s).
- High Precision: Sub-pixel registration reached a maximum error of approximately 0.03 pixels, ensuring that even minute movements are corrected to maintain visual stability.
Conclusion
This paper demonstrates that high-precision 2D registration can effectively solve the flicker problem in video face replacement without the need for complex 3D modeling. The analytic sub-pixel registration method provides a significant leap in both speed and accuracy. Future work may involve extending this analytic approach to handle more complex 3D deformations or extreme head poses where simple 2D registration might reach its limits.
Summary updated