Abstract
This paper introduces the Spatial-Temporal Nested GAN (STN-GAN), a framework designed for realistic face inpainting in anonymized videos. By integrating 3D residual blocks into a spatial inpainting architecture, the model ensures high temporal consistency and realistic facial synthesis. STN-GAN utilizes facial landmarks as soft constraints to maintain pose and expression while decoupling identity to preserve privacy. Experimental results on CelebA, 300-VW, and FaceForensics datasets demonstrate that STN-GAN significantly outperforms state-of-the-art methods in both quantitative metrics and human preference, achieving a Fréchet Inception Distance (FID) of 6.02 on the 300-VW dataset.
Introduction
In sensitive domains such as healthcare and security, privacy regulations often mandate the anonymization of facial data. Common techniques, such as masking the eyes, nose, and mouth, render datasets nearly useless for downstream tasks like human pose estimation or activity recognition that require full facial context. The challenge of "usability recovery" involves filling these masks with realistic, temporally smooth features without recovering the original subject's identity. Existing image inpainting methods often produce flickering artifacts in video, while video-to-video synthesis models typically require prohibitively large datasets to train effectively.
Methodology
STN-GAN addresses these challenges through a progressive architecture built on three pillars:
1. Spatial Inpainting with "Hard Copy"
The base model is a GAN trained with a combination of adversarial loss (WGAN-GP) and perceptual loss via a pre-trained VGG-16. To ensure seamless blending with the original background, the system employs a "background hard copy" technique, directly stitching non-masked pixels onto the generated output to prevent color or texture shifts.
2. Spatial Constraints via Landmarks
Rather than unconstrained generation, the model incorporates facial landmarks as an additional input. These landmarks act as soft constraints, allowing the network to synthesize faces that accurately match the subject's head pose and expression.
3. Temporal Nesting
The core innovation, or "secret sauce," is the nesting of 3D Residual Blocks within the generator's decoder and the discriminator's layers. These blocks link a sequence of $T$ frames, enabling the network to learn temporal correlations. The model leverages transfer learning by initializing spatial layers with weights from an image-trained model and fine-tuning temporal parameters on smaller video datasets.
Experiments and Results
The researchers evaluated STN-GAN using CelebA for initial spatial training, and 300-VW and FaceForensics for temporal evaluation.
Quantitative and Qualitative Performance
STN-GAN significantly outperformed existing methods. On the 300-VW dataset, it achieved an FID of 6.02, a substantial improvement over PartialConv (13.92) and Vid2Vid (39.61). In a user study, STN-GAN was preferred 84.31% of the time, primarily due to its ability to eliminate the "jitter" and flickering common in frame-by-frame inpainting methods.
Key Insights
- Efficiency: Transfer learning allows the model to achieve high temporal consistency without requiring massive video datasets.
- Controllability: Landmarks enable "parameterized inpainting," allowing users to perturb expressions (e.g., opening a closed mouth) while maintaining video integrity.
- Privacy: The model is designed to decouple identity, synthesizing realistic but "fake" features to preserve the original subject's privacy.
Conclusion
STN-GAN provides a robust solution for recovering the usability of anonymized video data. While the model currently depends on the accuracy of landmark detectors and is restricted to research use, it demonstrates a clear path forward for privacy-preserving data synthesis. Future work may explore adapting the framework for tasks where identity preservation is required or improving robustness against extreme head poses.
Summary updated