TL;DR
This paper introduces a novel, lightweight framework for generating high-fidelity co-speech gesture videos by using 2D full-body skeletons as an auxiliary condition to bridge audio and visual information. To address the scarcity of large-scale datasets, they also release CSG-405, a comprehensive dataset that aims to democratize research in this area.
What Problem Does This Paper Address?
Generating realistic videos of people speaking while also performing natural, synchronized gestures is a challenging task. The core difficulty lies in the "one-to-many" mapping problem: a single piece of audio can correspond to a vast range of possible gestures and expressions. This complexity is exacerbated by the need for high computational resources and the lack of large, diverse public datasets, which have historically limited research progress. Existing methods often struggle with artifacts, especially during significant pose changes, or rely on private datasets, hindering broader academic exploration.
Methods
The proposed framework consists of two main stages:
- Audio-to-Skeleton Prediction: This stage uses a diffusion model, specifically a Diffusion Transformer (DiT), to generate a sequence of 2D full-body skeletons from audio input.
- Key Innovation: The model is conditioned on both the audio and a skeleton extracted from the reference image of the target speaker. This ensures the generated skeletons are consistent with the speaker's body shape.
- Synchronization: To enforce strict audio-gesture coordination, the model concatenates embeddings of fine-grained audio segments and skeleton sequences along the feature dimension, rather than relying on less precise cross-attention mechanisms.
- Local Motion Representation: For improved realism, the model utilizes local motion representations for keypoints on the face and hands, disentangling global motion from subtle facial and hand movements.
- Human Video Generation: The predicted skeleton sequence, along with the speaker's reference image, is then fed into an off-the-shelf human video generation model to synthesize the final co-speech gesture video.
Datasets
- Training and Evaluation: The authors evaluate their method on existing datasets:
- PATS [1]
- TED-talks [6]
- New Dataset: The paper introduces CSG-405, a large-scale, publicly available dataset designed to democratize research.
- Size: 147,550 clips, totaling 405 hours of video.
- Content: Covers 71 diverse speech types, with high-resolution (512x512) videos.
- Annotations: Includes 2D skeleton annotations (133 keypoints) for each frame, along with diverse speaker demographics (gender, ethnicity, age, emotion) and speech scenarios (formal/casual).
- Quality Control: Videos undergo multi-stage filtering for skeleton quality, audio-lip sync, and speaker identification.
Key Insights and Ideas
- 2D Skeletons as a "Sweet Spot": The paper highlights that 2D full-body skeletons offer an efficient and effective auxiliary condition. They are more concise than raw visual data but rich enough to guide realistic gesture generation, bridging the gap between audio and visual outputs.
- Reference Image for Body Shape Consistency: Conditioning the skeleton generation on the reference image's skeleton is crucial. This simple yet effective step ensures that the generated gestures align with the target speaker's inherent body shape, preventing unnatural poses.
- Feature-Level Fusion for Synchronization: Concatenating audio and skeleton embeddings directly, rather than using cross-attention, leads to better temporal synchronization. This suggests that a more direct fusion of these modalities is key for accurate co-speech gesture generation.
- Democratization through Data: The release of CSG-405 is a significant contribution. By providing a large, diverse, and well-annotated dataset, the authors aim to lower the barrier to entry for researchers in this complex field.
Key Results
- Quantitative Performance: The proposed method achieves state-of-the-art results on the unpaired setting across PATS, TED-talks, and their own CSG-405 dataset, outperforming existing methods in metrics like FID and FVD (indicating better visual quality and realism) and Sync-C/Sync-D (indicating better audio-visual synchronization).
- Qualitative Improvements: Visual comparisons demonstrate that the method produces more natural, synchronized gestures with fewer artifacts compared to warping-based methods and methods that rely on external skeleton conditions. The generated body shapes are more consistent with the reference images, especially in unpaired settings.
- Generalization: The framework demonstrates strong generalization capabilities across different speakers and contexts.
Limitations and Future Work
- Limitations:
- The paper notes that in the paired setting, methods that directly use ground-truth skeletons (like StableAnimator) can achieve slightly better SSIM and PSNR. This is attributed to the randomness introduced by generating skeletons from audio, which is necessary for unpaired scenarios but can slightly reduce pixel-level fidelity compared to using perfect ground truth.
- The EchoMimicV2 model, when used as a generator, sometimes produces videos with the speaker positioned centrally and facing forward, which can be misaligned with the original co-speech gesture video.
- Future Work:
- The authors suggest that their CSG-405 dataset can be used to further enhance the performance of existing human video generation models.
- The paper implicitly points towards further research in more robust audio-to-skeleton prediction and more sophisticated video generation models that can better leverage auxiliary conditions.
- The authors also advocate for the development of ethical frameworks for synthetic content, suggesting a broader societal impact consideration for this technology.
Summary updated