Introduction
Generating realistic human motion that synchronizes naturally with speech is a cornerstone of creating believable digital avatars. While significant progress has been made in "talking head" synthesis—focusing primarily on lip-syncing—the generation of co-speech body gestures remains a significant challenge. Existing methods often fall into two categories: generative models that produce every frame from scratch, often leading to "texture mush" and temporal flickering, and retrieval-based models that stitch together real video clips, which frequently suffer from jarring transitions.
TANGO (Co-speech Gesture Video Reenactment) introduces a hybrid framework that combines the visual fidelity of retrieval-based methods with the temporal smoothness of diffusion-based generative models. By leveraging a hierarchical audio-motion embedding and a consistency-focused interpolation module, TANGO produces high-fidelity, audio-synchronized videos that outperform previous state-of-the-art approaches.
The Challenge of Co-Speech Gestures
The difficulty in co-speech gesture generation lies in the complex relationship between audio and motion. Gestures are not merely reactive to individual sounds; they follow both the rhythmic "beats" of speech and the broader semantic "vibe" of the conversation.
Generative models, while capable of following audio cues, often struggle with the fine details of human appearance, such as the intricate movements of fingers or the texture of clothing. Conversely, retrieval-based methods like GVR preserve these textures by using real video segments but struggle to find clips that perfectly match the timing of new audio, leading to "glitchy" jumps at the boundaries where clips are joined.
The TANGO Framework
TANGO addresses these issues through a three-stage pipeline: Graph Construction, Hierarchical Alignment (AuMoCLIP), and Appearance Consistent Interpolation (ACInterp).
Graph Construction and Pruning
The process begins by representing a reference video as a "motion graph." Nodes in this graph represent short, high-quality video clips, while edges represent viable transitions between them. To ensure the system can generate video of arbitrary length without hitting a "dead end," the authors employ Strongly Connected Components (SCC) to prune the graph, keeping only the most robustly connected segments.
AuMoCLIP: Hierarchical Audio-Motion Embedding
To solve the alignment problem, TANGO introduces AuMoCLIP, a joint embedding space for audio and 3D motion. Inspired by OpenAI's CLIP, it aligns audio features (extracted via Wav2Vec2 and BERT) with motion features at two distinct levels:
- Low-level Alignment: Matches the rhythmic beats of speech to specific motion frames, ensuring the "micro-timing" is correct.
- High-level Alignment: Matches the overall semantic context of a speech segment to a gesture clip, capturing the "macro-context."
ACInterp: Diffusion-Based Interpolation
When retrieved clips do not align perfectly, TANGO uses ACInterp (Appearance Consistent Interpolation) to bridge the gap. Built on the AnimateAnyone architecture, this diffusion-based model "hallucinates" the missing motion between two clips. To maintain visual identity and stability, it utilizes a "Reference Motion Module" and "Homography Background Flow," preventing the person's appearance from shifting and the background from jittering during transitions.
Datasets and Evaluation
The performance of TANGO was validated on two primary benchmarks:
- Show-Oliver: A 26-hour dataset featuring talk-show host John Oliver. This is a notoriously difficult dataset due to complex studio backgrounds and irregular camera movements.
- YouTube Business: A new dataset introduced by the authors, featuring 12 speakers in professional presentation settings, providing a benchmark for high-quality, "clean" video generation.
Key Results
TANGO demonstrates a significant improvement over existing methods in both quantitative and qualitative metrics. On the Show-Oliver dataset, TANGO achieved a Fréchet Video Distance (FVD) of 1.379, substantially lower than the previous retrieval-based leader, GVR (1.615), and generative models like SDT (2.239).
In user studies, TANGO's results were nearly indistinguishable from ground truth video, with participants preferring TANGO 36.9% of the time compared to 37.6% for the original footage. Furthermore, AuMoCLIP's low-level retrieval accuracy reached 65.68%, a massive leap from the 35.38% achieved by previous onset-based methods.
Conclusion and Future Work
TANGO successfully bridges the gap between high-fidelity video retrieval and smooth generative interpolation. While it currently relies on a linear pose assumption for transitions and external tools for lip-syncing, it sets a new standard for realistic co-speech gesture generation. Future work aims to extend this framework to more diverse human motions, such as dancing or sports, where the interplay between audio and movement is equally critical.
Summary updated