Abstract
ReSyncer is a unified framework designed to achieve high-fidelity lip-syncing, speaking style transfer, and face swapping within a single model architecture. By "rewiring" a traditional Style-based generator to accept 3D facial mesh guidance, the system produces stable, realistic, and person-specific facial performances. ReSyncer effectively bridges the gap between generalized models that lack detail and personalized models that lack flexibility, outperforming existing state-of-the-art methods in both visual quality and synchronization accuracy.
Introduction
Generating realistic videos of a person speaking based on an audio track—known as lip-syncing—is a cornerstone of digital human creation. However, the field faces a significant "cross-domain" challenge: mapping low-dimensional audio signals into high-dimensional, high-resolution video pixels without falling into the "uncanny valley."
Existing models typically fall into two categories: generalized models, which often produce blurry results or generic speaking styles, and personalized models, which require extensive training data for each specific individual. Furthermore, most models are specialized for a single task. ReSyncer addresses these limitations by providing a versatile framework capable of multiple facial reenactment tasks, including identity swapping and style transfer.
Methodology
The ReSyncer pipeline consists of two primary stages that transition from audio to 3D geometry, and subsequently from geometry to high-fidelity pixels.
Stage 1: Style-SyncFormer
To avoid the artifacts associated with direct audio-to-video mapping, ReSyncer first predicts 3D facial dynamics. The Style-SyncFormer is a Transformer-based module that processes audio features (extracted via Wav2Vec2) and a "style template" (a short sequence of 3D meshes from a source person). It outputs stylized 3D mesh displacements, enabling "zero-shot" speaking style transfer—mimicking the unique mouth movements of one person on the face of another.
Stage 2: Rewiring the Style-based Generator
The core innovation of ReSyncer is the reconfiguration of a Style-based generator (similar to StyleGAN) to utilize geometric guidance.
- Spatial Guidance: A white-colored 3D mesh is overlaid onto the masked target frame, providing the generator with a precise geometric "map" of the lips and jaw.
- Information Injection: Texture and identity information from a reference frame are injected into the generator's style space ($W$), while dynamic motion from the 3D mesh is injected into the noise space and feature maps.
This "rewiring" allows the model to prioritize geometric structure while maintaining high-resolution textures.
Experimental Setup
ReSyncer was trained and evaluated on several large-scale datasets:
- HDTF & VoxCeleb2: Used for general lip-syncing training across thousands of "in-the-wild" videos.
- FaceForensics++ (FF++): Utilized to evaluate face-swapping performance.
- Personalized Benchmarks: High-resolution video sets (e.g., "Trump" and "Taylor Swift") demonstrated the model's ability to be fine-tuned on minimal footage.
Discussion and Key Insights
- Geometric Guidance: Using a simple, white-colored 3D mesh provides a "Goldilocks" level of detail—more spatial information than raw coefficients but less complexity than fully textured rendering.
- Attribute Separation: The framework successfully decouples speaking style (motion) from identity (appearance). By injecting these into different parts of the network, ReSyncer can mix and match attributes seamlessly.
- Unified Architecture: The reliance on geometric guidance allows the same model to handle both lip-syncing and face-swapping by simply adjusting the mask and reference identity.
Results
ReSyncer consistently outperforms state-of-the-art (SOTA) methods like Wav2Lip and StyleSync.
- Lip-Sync Accuracy: On VoxCeleb2, ReSyncer achieved a Landmark Distance (LMD) of 3.19, surpassing StyleSync (3.28) and Wav2Lip (3.44).
- Visual Quality: In user studies, ReSyncer's "Video Realness" was rated 4.54/5.0, significantly higher than StyleSync's 3.11.
- Face Swapping: The model achieved an identity retrieval rate of 96.7% on FF++, demonstrating performance comparable to specialized face-swapping models.
Conclusion
While ReSyncer represents a significant advancement, it faces challenges with extreme head rotations (over 80 degrees) due to limitations in current 3D reconstruction tools. Future work will focus on improving robustness for side profiles and integrating expressive emotional controls. Given the high fidelity of the results, the authors emphasize ethical considerations and the importance of restricting the technology to research purposes to prevent misuse.
Summary updated