Motion-Prompted Recurrent Gesture Generation for Unseen Speakers

Systems that animate a body from speech, so-called co-speech gesture generators, are trained either on one performer, as with Trinity, or on a fixed roster of enrolled speakers, as with BEAT2's 25 actors. A new person therefore gets either a borrowed style or a generic "average gesturer." The fixes in the literature are all expensive at the moment of use. One looks up identity in a table of known speakers. Another retrieves poses from a motion database recorded for that specific person. A third fine-tunes the network on minutes of their data. The method most suited to live, frame-by-frame animation on a robot or avatar is a small recurrent network, meaning a network that carries a running memory vector forward one frame at a time. These networks have the opposite problem: over long sequences their own errors feed back into them, and the motion drifts, freezes, or collapses to a neutral pose.

We propose one mechanism that addresses both problems. A few seconds of ordinary video of the new person, converted to a skeleton by an off-the-shelf pose estimator (software that detects joint positions in video), serves two roles. It is a style prompt for a causal recurrent generator, and it is a set of speaker-specific "home poses." The generator's memory is pulled back toward these home poses whenever it starts to drift. The speech audio is first stripped of voice identity, so it contributes only rhythm and content. The person's style comes from how they move, not from how they sound.

Core Claim

Gesture style and anti-drift anchoring can share a single source: a short, pose-estimated motion clip of the target person. A recurrent generator conditioned this way, and fed speaker-normalized audio, should do three things. It should reproduce the style of speakers never seen in training, with no weight updates and no per-person database. It should stay stable over minute-long streams. It should run with frame-level latency.

The novelty is not "a GRU with a speaker embedding." That much exists. Two things are new. The first is that style is drawn from a motion prompt rather than an ID or a voice. The second is a gated correction of the recurrent state that uses the prompt's own poses as anchors. That correction replaces StreamTalk's retrieve-and-inpaint loop with something a recurrent network can execute in constant time per frame.

Where This Sits

Three recurrent multi-speaker generators define the starting point. Speech gesture generation from the trimodal context of text, audio, and speaker identity maps one-hot speaker IDs into a learned style space and decodes with a bidirectional GRU. BEAT uses fixed actor IDs with separate LSTM branches for body and hands. Speech-Gesture GAN feeds raw speech features (MFCCs) into a Bi-GRU and trains only on the single Trinity speaker. The first two cannot represent a person outside their ID table. All three bidirectional or offline designs read future audio, which rules out streaming. We keep their recurrent decoder, including BEAT's split of body and hands, but make it causal and replace IDs with a prompt encoder.

Three non-recurrent lines motivate the prompt and the anchor. StreamTalk shows that pulling a drifting stream back to a valid pose keeps gesture error stable over 60 seconds. It does this with a Diffusion Transformer that works on 60-frame chunks and retrieves anchors from a database. That database works best when it is speaker-specific and dilutes style when it falls back to a global pool. We keep the idea of a corrective anchor but take the anchors from the prompt and apply them inside the recurrent state. Democratizing High-Fidelity Co-Speech Gesture Video Generation conditions a diffusion model on a single reference skeleton to fix body shape. We extend a static reference to a moving prompt that carries style, not just proportions. MambaGesture separates style and emotion from audio with a self-supervised module. It is evaluated only on enrolled speakers, and it relies on iterative diffusion.

Two papers supply methods we reuse. Transformer-S2A separates speaker-independent phonetic content from pitch and energy for face animation. It does this with phonetic posteriorgrams, which are per-frame phoneme probabilities taken from a speech recognizer. We reuse this audio front end for the body.

Two papers constrain the design. Style Transfer for Co-Speech Gesture Animation reports that a single generator trained on pooled speakers collapses to an average style: Inception Score 2.06, against 4.48 for its mixture model. So a prompt that the decoder can ignore will be ignored, and the training must force the decoder to use it. Continual Learning for Personalized Co-Speech Gesture Generation shows that per-speaker fine-tuning forgets catastrophically unless it is carefully regularized. That result motivates avoiding weight updates altogether.

Research Program

The model

The generator is a causal two-branch GRU over body and hand joint rotations. It allows a configurable audio lookahead of 0 to 200 ms. Audio enters as phonetic posteriorgrams plus pitch and energy normalized per utterance, so voice identity has little route into the motion.

A prompt encoder reads 5 to 20 seconds of the target's skeleton and produces two outputs. The first is a style vector, which modulates the GRU through feature-wise scaling and shifting (FiLM, as used for pose feedback in Speech-Gesture GAN). The second is a small set of $K$ prototype poses, obtained by clustering the prompt frames.

At each frame the state is corrected as

$$h_t \leftarrow h_t + g_t\,\big(E(p^{*}_t) - h_t\big),$$

where $h_t$ is the GRU memory and $p^{*}_t$ is the prototype nearest the current predicted pose. $E$ is a learned map from poses into memory space. $g_t \in [0,1]$ is a gate computed from the distance between the recent predicted poses and the prompt's pose distribution. The gate stays near zero during normal motion and opens only when the output wanders away from what this person does.

Training

Training uses speaker-disjoint prompt and target clips from the same person. A contrastive term makes the style vector identify the person. A prompt-swap term penalizes outputs whose style statistics match the audio's speaker rather than the prompt's speaker; this is how we force the decoder to use the prompt. To make the gate learn correction, we train with scheduled exposure to the model's own rollouts. To mimic video-based pose estimation, prompts are corrupted with joint jitter and dropout that is heaviest on the fingers. This reflects the finger-tracking noise that forced EMAGE to drop roughly 16 hours of BEAT2.

Analysis

Beyond accuracy, we measure how often the gate opens and whether it opens before drift becomes visible. We also probe whether the style vector or the memory carries identity, using speaker classifiers trained on frozen features.

Baselines

Naive reference points. There are two. The first is a mean-pose output. The second is a speaker-agnostic causal GRU with the same audio front end and no prompt. These two set the floor for style metrics and show how much identity any method actually recovers.

Strongest existing method for quality. This is GestureLSM, a flow-matching generator. It reports FGD 4.088 on BEAT2 and 0.039 s per frame. FGD, the Fréchet Gesture Distance, measures how far the distribution of generated motion is from real motion; lower is better. GestureLSM is the strongest competitor because it is both accurate and fast. We will retrain it under our held-out-speaker split with the same audio features and a comparable tuning budget, giving it a prompt-derived style input so the comparison tests architecture and not access to information.

Strongest method for anchoring. We reimplement the retrieve-and-refine loop of StreamTalk, which reports 76 FPS on a V100. No numerical FGD for it was found in the file. We run it in two forms: with a global database, and with an oracle database holding all of the held-out speaker's data. The oracle form is an upper bound that our prompt-only anchoring would have to approach.

Recurrent reference. CaMN from BEAT reports FGD 65.74 on BEAT in the MambaGesture benchmark, where MambaGesture reaches 22.11. This is the recurrent reference, made causal for the streaming comparisons.

Ablations. These isolate the key ingredients. We remove the gate. We use anchors drawn from global rather than prompt prototypes. We use the prompt for style only, without anchoring. We replace the motion prompt with a voice embedding.

Beating the baselines means two things. First, on unseen speakers, style identification and FGD should match or beat GestureLSM at strictly lower algorithmic latency. Second, 60-second drift should be within the oracle-database StreamTalk's range. Because protocols differ, we will compare only numbers we rerun ourselves; published numbers are anchors, not targets.

Why not just…?

Why not just concatenate a pretrained speaker-voice embedding to the GRU input? This is the cheapest zero-shot route and the most serious threat. It is our "voice embedding" ablation. Our hypothesis is that voice predicts gesture style only weakly. MambaGesture gains from disentangling style rather than concatenating it, but only on enrolled speakers. The prompt-swap test settles the question: pair speaker A's audio with speaker B's motion prompt, and see whether the output style tracks B. If a voice embedding matches the motion prompt on held-out style identification, we will report that motion prompts are unnecessary and drop that claim.

Why not just use a fast non-recurrent model with a style prompt? GestureLSM's 0.039 s per frame is already real-time in throughput. The objection to it is lookahead. Chunked models, such as StreamTalk's 60-frame windows, must buffer future audio. The latency sweep from 0 to 200 ms will show whether that matters. If GestureLSM with a prompt matches us at equal lookahead, the recurrent claim reduces to a parameter-count argument, and we will say so.

Why not just fine-tune on a few minutes of the new speaker? C-DiffGAN shows this can work with 2 to 10 minutes of data, but it needs careful regularization against forgetting. We include fine-tuning as an upper-bound comparison at matched data: 20 seconds versus several minutes. We expect it to win given enough data. The question is how much of its gain a gradient-free prompt recovers.

Proposed Experiments

Data. Our main benchmark is BEAT2, using leave-five-speakers-out folds. Prompts come from held-out recordings of the test speaker and are noise-corrupted to resemble pose estimation. To test prompts that genuinely come from video, we train and test on PATS, whose 2D skeletons were extracted from talk-show video, again with disjoint speakers. The TED Gesture dataset's 391 speakers test breadth.

Metrics. We report FGD, beat alignment, and diversity. Style fidelity is the top-1 accuracy of a classifier, trained on real held-out-speaker motion, at identifying the prompted speaker. Drift is the per-window distance of pose statistics over 60-second rollouts. We also report per-frame compute and algorithmic latency, plus a user study pitting prompted-speaker output against a mismatched speaker.

Decisive checks. There are three. The first is the prompt-swap test: style must follow the motion prompt, not the voice. The second is the gate ablation: removing the gate must cause measurable drift by 30 to 60 seconds, while prompt anchors must beat global anchors on style fidelity. The third is a sweep of prompt length and prompt noise, showing how many seconds of noisy video skeleton are enough. Failing the first check means identity is leaking through audio. Failing the second means the anchoring mechanism is decorative.