TL;DR
This paper introduces CoCoGesture, a novel framework for generating realistic and diverse 3D co-speech gestures from raw human audio. By leveraging a massive new dataset (GES-X) and a pre-train/fine-tune strategy with a large diffusion model and a unique Mixture-of-Gesture-Experts block, CoCoGesture achieves state-of-the-art performance in zero-shot speech-to-gesture generation.
What Problem Does This Paper Address?
Generating realistic 3D gestures that synchronize with human speech is crucial for creating lifelike virtual avatars and enhancing human-computer interaction. However, existing methods struggle with generating diverse and natural gestures, especially when faced with unseen human speech. This limitation stems primarily from the scarcity of large-scale, high-quality 3D co-speech gesture datasets. Current datasets are often small, lack diversity in speakers and gestures, and may rely on text transcripts or speaker IDs as input, hindering generalization to real-world "in-the-wild" scenarios. The challenge is to create a model that can produce coherent, expressive, and varied gestures directly from raw audio, even for speakers and speech patterns it hasn't encountered during training.
Methods
CoCoGesture employs a two-stage "pre-train, fine-tune" paradigm:
- Pre-training (Gesture Expert): The core idea is to train a large, generalizable gesture generation model first. The authors leverage a diffusion transformer backbone, scaled up to 1 billion parameters, and pre-train it on their massive GES-X dataset. This large model acts as a "gesture expert," learning the manifold of human postures and dynamics. This pre-training phase is crucial for achieving robustness and generalization.
- Fine-tuning (Audio ControlNet & MoGE): For fine-tuning, they introduce an "Audio ControlNet." This network incorporates the human speech audio as a conditioning signal. It uses a trainable copy of the pre-trained diffusion model. The key innovation here is the Mixture-of-Gesture-Experts (MoGE) block. MoGE adaptively fuses audio embeddings from the speech with gesture features from the pre-trained gesture expert. A novel routing mechanism within MoGE dynamically balances these inputs, ensuring temporal coordination between audio and motion while preserving gesture diversity. This allows the model to generate gestures directly from audio without relying on intermediate text representations.
Datasets
- GES-X (New Dataset): The paper introduces GES-X, a large-scale, whole-body 3D meshed co-speech gesture dataset. It contains over 40 million meshed posture instances from approximately 4.3K speakers, extracted from in-the-wild talk show videos. This dataset is significantly larger than existing benchmarks and aims to provide a more uniform distribution of gesture motions.
- BEAT2 & TalkSHOW: These existing datasets are used for evaluation. BEAT2 provides 3D meshed whole-body postures with multi-modality information (speaker ID, text), while TalkSHOW focuses on sitting postures and lacks text transcripts, making it a good testbed for zero-shot audio-only generation.
Key Insights and Ideas
- Pre-training is Key for Generalization: Training a massive gesture diffusion model on a large dataset (GES-X) first is essential for creating a robust "gesture expert" that can generalize to unseen audio and speakers.
- Direct Audio-to-Gesture is More Practical: Bypassing text transcripts and directly conditioning gesture generation on raw audio is a more robust and practical approach for real-world applications, especially in noisy environments.
- Mixture-of-Gesture-Experts (MoGE) for Adaptive Fusion: The MoGE block, with its adaptive fusion and routing mechanism, is critical for effectively integrating audio information with learned gesture priors, leading to temporally coherent and diverse outputs.
- Scale Matters: Scaling up the diffusion model to 1 billion parameters significantly enhances its ability to learn complex gesture manifolds.
Key Results
- State-of-the-Art Performance: CoCoGesture outperforms existing methods on both BEAT2 and TalkSHOW datasets across key metrics like Fréchet Gesture Distance (FGD), Diversity, and Beat Alignment (BA).
- Zero-Shot Superiority: Notably, on the TalkSHOW dataset (which is out-of-domain and lacks text transcripts), CoCoGesture achieves a significant reduction in FGD (16.22% over suboptimal counterparts), demonstrating its strong zero-shot generalization ability.
- Improved Diversity and Coherency: Qualitative results and user studies show that CoCoGesture generates more vivid, diverse, and rhythmically aligned gestures compared to other methods.
- Dataset Impact: The GES-X dataset is shown to be crucial for the pre-training stage, leading to better generalization than models pre-trained on smaller datasets like BEAT2.
Limitations and Future Work
- Audio-Only Input: The current framework only takes audio as input. The authors acknowledge that this might lead to emotionally insensitive gestures (e.g., not reflecting anger or happiness appropriately).
- Automated Pose Extraction Impact: The use of automated pose estimation and speech techniques for dataset creation might introduce some noise or inaccuracies, despite extensive filtering.
- Future Work: The authors suggest incorporating emotional conditions into the model and exploring more robust data processing techniques to further improve gesture quality. They also plan to release the GES-X dataset to facilitate future research.
Summary updated