Systems that animate a talking digital human's whole body have split into two camps. Fast systems drive the face and body from speech audio alone and run in a few sampling steps. Expressive systems also read the transcript, so a speaker who says "over there" can point and one who says "I don't know" can shrug. The transcript costs latency. A streaming speech recognizer only commits to a word after the word has been spoken, and a language encoder adds compute to every frame. This proposal asks whether the transcript can serve as a teacher during training only and then be thrown away. At test time the avatar would hear nothing but audio and would still produce gestures that reflect the meaning of the speech.
The idea comes from scene text recognition. SVTRv2 adds a Semantic Guidance Module during training that uses surrounding character context to shape the visual features. It removes the module at inference and keeps a fast one-pass decoder. The module is worth a 5.11% accuracy gain on occluded text, and the resulting model is 8× faster than MAERec. We carry this recipe over to audio-driven whole-body motion. A transcript-reading teacher branch pulls the body branch's audio features toward the ones that meaning-aware gesturing needs. Only the audio branch is deployed, sampled in a few steps with a consistency model (a generator distilled so that one or a few denoising steps replace dozens).
Core Claim
A few-step, dual-branch avatar generator that sees only audio at inference can recover most of the semantic-gesture quality of a transcript-conditioned model, at no added runtime cost, if two conditions hold. First, during training its body-branch features are distilled toward those of a transcript-attending teacher. Second, the distillation is weighted toward the frames where gesture carries meaning. The face branch is kept out of this guidance, because lip motion depends on phonemes rather than semantics. The claim can be falsified in two ways. It fails if the guided audio-only model does no better than an identical unguided model on semantically weighted metrics. It also fails if a plain speech-recognition auxiliary loss on the audio features does equally well. In that second case the simpler method is the answer and we will report it as such.
Relation to Prior Work
The fast camp is best represented by AsynFusion. It uses separate face and body diffusion-transformer branches that exchange information through bidirectional cross-attention, a module the authors call CoSync. Each branch is sampled with its own number of consistency-model steps, which cuts batch inference from 56.4 s to 15.9 s on BEAT. Hogue et al. similarly share one small 27.7M-parameter denoiser between face and body through cross-modal adapters. Both are conditioned only on acoustic features. They motivate this proposal because they show that coordinated, fast face-body generation works. They also leave out language entirely.
The expressive camp uses language at runtime. EMAGE merges prosody with spoken content through a content-rhythm attention module inside an iterative masked transformer. BEAT's CaMN baseline concatenates text, audio, and other streams through convolutional and recurrent layers. The humanoid-robot system of Zhang has a large language model read the transcript and retrieve gestures from a library. We keep their insight that body gesture follows semantic intent, as Fengqi Liu et al. also argue. We move the language pathway entirely into training.
TiCLS constrains the design. Fusing a character language model into a text spotter by cross-attention costs 2.73× the parameters and 1.55× the latency, which is the overhead we refuse at runtime. TiCLS also failed to converge when its language decoder was randomly initialized, so our teacher uses a frozen pretrained text encoder.
MuEx shows that lip motion is best grounded in phoneme-level articulatory prototypes. That finding supports keeping semantic guidance away from the face. Finally, Mafla's out-of-vocabulary observations warn that language priors can make a model "lazy," so that it produces stereotyped outputs for frequent words. We therefore test gesture diversity and behavior on rare words explicitly.
Research Program
The student is a dual-branch few-step generator in the style of AsynFusion. It has separate face and body branches, bidirectional face-body cross-attention after each block, and asynchronous consistency sampling. It is conditioned on audio features from a pretrained speech encoder, with a small fixed look-ahead window for streaming.
During training we attach a teacher pathway to the body branch. At each block, audio frame features act as queries into time-aligned word embeddings from a frozen text encoder, using the cross-attention form surveyed in the hierarchical video summarization work. The attended context enters through a learned gate, following Yeachan Kim et al., so that semantic context cannot wash out the rhythmic audio stream. The teacher shares all other weights with the student. Both produce motion, and the training loss is
$$ \mathcal{L} = \mathcal{L}_{\text{gen}}^{s} + \mathcal{L}_{\text{gen}}^{t} + \lambda \sum_{\ell} \sum_{f} w_f \, \lVert h^{s}_{\ell,f} - \mathrm{sg}(h^{t}_{\ell,f}) \rVert^2 . $$
Here $\mathcal{L}_{\text{gen}}^{s}$ and $\mathcal{L}_{\text{gen}}^{t}$ are the generation losses of the student and teacher. The terms $h^{s}_{\ell,f}$ and $h^{t}_{\ell,f}$ are their body-branch hidden states at layer $\ell$ and frame $f$, $\mathrm{sg}$ stops gradients into the teacher, and $\lambda$ sets the distillation strength. The weight $w_f$ is BEAT's frame-level semantic relevance score. It concentrates the distillation on frames where a gesture carries meaning rather than beat.
After this stage we run consistency distillation on the student alone to reach AsynFusion-level step counts. At inference the text pathway is removed, so the deployed network is architecturally identical to an unguided baseline.
We will also analyze what the student has learned. A linear probe on student body features will predict the words and the semantic-gesture category, measured with and without guidance. This shows whether the lexical information comes from the audio encoder or from the distillation. We will also measure how far teacher and student motion diverge as a function of look-ahead window size, which gives the latency-quality curve for streaming.
Baselines
The naive reference point is the do-nothing student. It is the same dual-branch audio-only architecture, trained with the same data, steps, and tuning budget, with no teacher. Because the deployed guided model has the same architecture and FLOPs, any gain over this reference comes from training-time guidance alone. Beating it means a clear improvement in semantic relevance gesture recall (SRGR, BEAT's recall metric weighted by semantic importance) computed on high-relevance frames, with no loss in FGD, FMD, or face-branch error.
The strongest existing fast method is AsynFusion. On BEAT it reports FMD of 312.46, FGD of 421.58, and a batch inference time of 15.9 s. We will re-train it under our split and feature extractor with a matched tuning budget, and cite the published numbers alongside. BEAT also reports CaMN at an FGD of 123.7. That figure is far below AsynFusion's, which almost certainly reflects a different FGD feature extractor or protocol. We will therefore not treat it as comparable and will re-evaluate CaMN under one protocol. We found no reported SRGR value for AsynFusion or EMAGE on Lacuna, so semantic comparisons will rest on our own re-runs.
The strongest semantic competitor runs the teacher itself at inference in two forms. One uses oracle word-aligned transcripts, which gives an upper bound. The other uses transcripts delayed by a realistic streaming-recognition lag, which is the honest deployed alternative. EMAGE, re-trained on BEAT2, provides an external runtime-text reference.
Two ablations isolate the key ingredient. The first uses uniform weights ($w_f = 1$), which tests whether relevance weighting matters. The second applies guidance to both branches, which tests whether semantic guidance disturbs lip motion.
Why not just…?
Why not just feed the transcript at runtime? This is the real threat when transcripts are available and a delay is tolerable, and the oracle-teacher baseline measures exactly how much it buys. The case for our approach rests on streaming. A word is only committed after it ends, while meaningful gestures tend to co-occur with or precede the word they relate to. A runtime-text model therefore either lags or needs a longer look-ahead window. Our delayed-transcript baseline turns this argument into a measurement. If the delayed-text model beats the guided student at equal end-to-end latency, the proposal's practical case fails.
Why not just use a stronger audio encoder? Large self-supervised speech encoders already carry some lexical content, so the guidance could be redundant. We hold the encoder fixed across the guided and unguided students and repeat the comparison with a second, larger encoder. If the guidance gain disappears with the larger encoder, that is a meaningful negative result: the semantics were already available in the audio, and the only need was a better encoder.
Why not just add a speech-recognition loss on the audio features? Predicting the transcript from student features is simpler, and SVTRv2's module is close to this. Our hypothesis is that recognition supervision shapes features toward phonetic identity, whereas the motion-predicting teacher shapes them toward gesture-relevant meaning. We will train this auxiliary-loss variant with the same budget. If it matches our method on semantic-frame SRGR, we will adopt it and report that the teacher is unnecessary.
Proposed Experiments
We train and evaluate on BEAT, which provides transcripts and frame-level semantic relevance for computing $w_f$, and we replicate on BEAT2 for the SMPL-X/FLAME representation. The metrics are FGD and FMD, SRGR over all frames and over high-relevance frames only, face-branch reconstruction error, and gesture diversity across repeated samples. Latency is reported as milliseconds per frame and streaming real-time factor on one GPU, together with parameter count.
Three checks are decisive. The first is whether the guided student closes a substantial fraction of the SRGR gap between the do-nothing student and the oracle teacher at identical inference cost. The second is whether it beats both the recognition-loss variant and the delayed-transcript model at matched latency. The third is whether it avoids Mafla-style laziness. For that check we compare SRGR and diversity on utterances containing rare words against those containing frequent words, looking for any collapse onto stereotyped gestures. Face-branch metrics must not degrade relative to the do-nothing student; if they do, keeping guidance body-only is shown to be necessary.
References
- SVTRv2: CTC Beats Encoder-Decoder Models in Scene Text Recognition
- AsynFusion: Towards Asynchronous Latent Consistency Models for Decoupled Whole-Body Audio-Driven Avatars
- Joint Co-Speech Gesture and Expressive Talking Face Generation Using Diffusion with Adapters
- EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modeling
- BEAT: A Large-Scale Semantic and Emotional Multi-modal Dataset for Conversational Gestures Synthesis
- Semantic Co-Speech Gesture Synthesis and Real-Time Control for Humanoid Robots
- Emphasizing Semantic Consistency of Salient Posture for Speech-Driven Gesture Generation
- TiCLS: Tightly Coupled Language Text Spotter
- A Bridge from Audio to Video: Phoneme-Viseme Alignment Allows Every Face to Speak Multiple Languages
- Hierarchical multimodal transformer to summarize videos
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention