Silence as a Domain Shortcut in Feed-Forward Audio-Driven Gaussian Avatars

Talking-head avatars built from 3D Gaussians (small colored, semi-transparent blobs that together render a face from any viewpoint) can now be driven by raw speech audio in real time. Almost all of them, though, are trained separately for each person. The newer "feed-forward" systems avoid that: they build an avatar of a person never seen in training from a few photos in under a second, but they are driven by tracked facial expressions rather than by sound. This note proposes a feed-forward, audio-driven avatar and argues that joining these two capabilities creates a failure that per-person models mostly hide. A model that has to generalize to new people and new microphones will read the background hiss of an unfamiliar recording as speech. It will then move the mouth during pauses. Speech recognition, spoofing detection, and keyword spotting have each run into the same problem.

The core claim is that silence handling is not cosmetic post-processing for these systems. It is a generalization problem, and it needs a specific fix: an asymmetric, learned speech gate that turns off the audio-driven mouth branch during non-speech and hands control to a separate audio-free "idle" motion model. The gate opens slightly before acoustic onset, because lips move before sound comes out. We expect this to beat both a simple energy or voice-activity cutoff and the default of letting attention learn to ignore silence, and we expect the gap to grow on unseen identities and unseen recording conditions.

Where the Gap Is

Three audio-driven Gaussian avatar systems motivate the problem, and none of them filters audio. GaussianSpeech rigs roughly 30–35k Gaussians to a face mesh and decodes mesh offsets from continuous wav2vec 2.0 speech features. SyncGaussian fights head jitter with better tracking and a two-stream speech encoder, but that encoder still feeds features through pauses. GenSync shares one network across 10 identities using identity–audio cross-attention. All three are trained per identity or on a closed set of identities. Each can therefore memorize what its own subjects' silence sounds like. Audio-Driven Universal Gaussian Head Avatars (UniGAHA) is the closest to generalizing: it maps Wav2Vec features into a universal expression space with a diffusion model. However, it needs about 20 minutes of fine-tuning for each new identity, and its authors report minor expression drift during neutral intervals.

FastGHA supplies the method on the reconstruction side. It builds a canonical Gaussian head from 2–6 views in under a second, using image features supervised by a pretrained 3D geometry model (the Visual Geometry Grounded Transformer, VGGT). A small deformation network conditioned on expression codes of a standard parametric face model (FLAME) then animates the head at 62–128 FPS. FastGHA takes no audio. The nearest thing to "silence" in the talking-head literature, Silence is Golden, is an adversarial defense that makes diffusion animators ignore audio. It is not a reconstruction method.

On the audio side, the evidence both motivates and constrains the design. WhisperX shows that a purely attention-based speech model hallucinates repeated text on non-speech unless audio is first cut into speech chunks by voice activity detection (VAD, a model that marks which frames contain speech). That fix improved word error rate on TED-LIUM from 10.5 to 9.7. Hu et al. report that soft attention fails to suppress uninformative frames where hard masking succeeds. Wan et al. and Chettri et al. show that models learn to associate channel background or trailing silence with labels. This shortcut breaks when the recording channel changes, which is exactly the situation a feed-forward avatar meets on in-the-wild audio.

The same literature also records the main risk of the naive fix. Hard thresholds such as the −20 dB cutoff used in TartanAviation clip quiet consonants and trailing sounds. WhisperX's cascade cannot recover from boundary errors made by its VAD stage. A face adds a further problem that transcription does not have: lip and jaw motion leads the acoustic onset. A gate that is timed exactly to the sound will therefore snap the mouth open late.

Core Claim

We make two claims. First, in a feed-forward audio-driven Gaussian avatar, mouth motion during non-speech (idle jitter) is caused mainly by the channel reaching the audio encoder, not by the new identity. It should therefore grow with channel mismatch between training and test audio more than with identity mismatch. Second, the fix is a gate with three properties. It is learned jointly with the avatar. It is asymmetric in time, opening ahead of acoustic onset and closing with a short tail so that quiet consonants survive. When it is closed, it routes control to an audio-free idle model rather than freezing the face.

If a fixed VAD freeze or silence-augmented training performs as well, the claim is wrong. The experiments below are designed to find that out.

Research Program

The system has three parts. The canonical head is built exactly as in FastGHA: sparse views go into a multi-view transformer, which outputs canonical Gaussians plus a latent vector for each Gaussian. A speech branch replaces FastGHA's tracked expression input. A frozen wav2vec 2.0 encoder feeds a small transformer that predicts face-model expression codes, following the audio-to-expression mapping in GaussianSpeech and UniGAHA but without diffusion, so the whole pipeline stays feed-forward and streamable.

The new component is the gate. A lightweight frame classifier predicts a speech probability $g_t$ at each frame $t$ from a window of audio features running from $t-k_b$ to $t+k_f$. We use a forward look-ahead $k_f$ of roughly 100–200 ms, bought with a matching output delay, and a shorter backward context $k_b$. The expression code sent to the face deformation network is

$$e_t = g_t\, f_{\text{speech}}(a_{t-k_b:t+k_f}) + (1-g_t)\, f_{\text{idle}}(e_{t-1}, z_{\text{id}}),$$

where $f_{\text{speech}}$ is the audio-to-expression network acting on audio window $a$, and $f_{\text{idle}}$ is a small recurrent model. The idle model sees only the previous expression $e_{t-1}$ and an identity latent $z_{\text{id}}$, and it produces blinks, breathing, and slow relaxation toward a closed mouth. During training, $g_t$ is supervised with speech/non-speech labels from a pretrained VAD whose boundaries are shifted earlier by the measured lip-leading offset. At inference, $g_t$ is thresholded, which makes it a hard mask in the sense of Hu et al.: silent frames never reach $f_{\text{speech}}$. The rest of the model is trained with FastGHA's photometric and geometric losses, plus a lip-sync loss.

The analysis part of the program tests the mechanism, not just the output. We measure idle jitter as a function of two separate factors. Identity shift compares seen with unseen subjects. Channel shift uses clean studio audio versus audio with room noise, telephone band-limiting, or microphone hiss mixed into the pauses only, so that speech content stays fixed. If channel shift dominates, that supports the shortcut explanation. We also probe whether wav2vec features during silence carry information about which speaker or dataset the recording came from; if they do, that is the route by which silence leaks.

Baselines

The naive reference is the same feed-forward avatar with no gate, where wav2vec features flow continuously into $f_{\text{speech}}$. Every other system is compared against this one, and it defines how much idle jitter there is to remove.

The simple silence-aware baseline is a fixed −20 dB energy gate, as in TartanAviation, and separately an off-the-shelf VAD cut with the face frozen during non-speech, as in WhisperX's cut-and-merge. These use the same trained avatar and differ only at inference, so the comparison isolates the gate design. The proposal beats them if it matches their idle jitter while scoring better on lip sync at speech onsets and on quiet-consonant segments.

The strongest existing audio-driven Gaussian avatar on the relevant data is UniGAHA. It reports PSNR 27.37 and LSE-D 6.32 on Ava-256 (LSE-D measures the distance between audio and lip features, where lower means better sync). It gets these numbers after about 20 minutes of fine-tuning per identity, so it is an upper reference rather than a like-for-like competitor. The strongest feed-forward reconstructor is FastGHA, which reports PSNR 22.5 and SSIM 0.77 on Ava-256 few-shot reconstruction, but it is driven by expressions and not by audio. We will reproduce FastGHA with tracked expressions as a ceiling on visual quality, and run UniGAHA with its published protocol where code permits. Our target is to stay within about one PSNR point of the tracked-expression FastGHA ceiling, to reach LSE-D near UniGAHA's 6.32 with no per-identity fine-tuning, and to clearly reduce idle jitter relative to all baselines.

Two ablations isolate the key ingredients. A symmetric gate with no look-ahead tests the asymmetric timing. A gate that freezes the face instead of calling $f_{\text{idle}}$ tests the idle model. All variants share data splits, backbone, and number of training steps, and each gets the same hyperparameter search budget, with thresholds tuned on a held-out validation set.

Why not just…?

Why not just let attention learn to ignore silence? This is the default in GaussianSpeech, SyncGaussian and GenSync. WhisperX and Hu et al. are direct evidence that attention-only models do not reliably suppress non-speech. Those results, however, come from recognition and classification, not facial motion. The no-gate baseline under channel shift is the experiment that settles this for avatars. If its idle jitter on unseen channels matches the gated model's, the premise fails.

Why not just use a fixed energy threshold or off-the-shelf VAD and freeze the face? This is the most plausible threat, since it costs nothing to add. The limitations recorded in the file predict it will clip quiet consonants and, for faces specifically, open the mouth late because lips move before sound. It will also produce a frozen, lifeless face during long pauses. If the frozen-VAD baseline ties the proposal on onset sync and human preference, we would drop the learned gate and report the cheap fix as sufficient.

Why not just train with added silence and noise augmentation? Mixing noisy pauses into training could teach $f_{\text{speech}}$ to output a neutral expression on its own, with no gate at all. This is a real competitor to the claim that a separate gate is needed. We run it as its own arm, with an augmentation budget matched to the channel perturbations used in testing. We then evaluate both arms on held-out perturbation types the augmentation never saw. The gate should hold up on those unseen channel types, whereas augmentation should only cover the channels it was trained on.

Why not just smooth the predicted expressions over time? A low-pass filter reduces jitter everywhere, including real fast lip motion. We include a filter whose strength is tuned to match our idle jitter, and compare lip-sync scores during speech.

Proposed Experiments

The data are Ava-256, which has multi-view capture with audio and was used by FastGHA and UniGAHA, for training and for testing on unseen identities. We add Nersemble views for broader identity coverage. The multi-view GaussianSpeech speech-avatar dataset and HDTF serve as cross-dataset audio and identity tests. Both are cross-domain tests, and HDTF additionally tests reconstruction from monocular frames.

The metrics are PSNR, SSIM and LPIPS for image quality; LSE-D and Sync-C for lip sync (Sync-C is the matching confidence score, where higher is better); and idle jitter. Idle jitter is the mean frame-to-frame speed of mouth-region Gaussian centers during labeled non-speech spans, along with the rate of spurious mouth openings. We also report onset lag, the delay between ground-truth and rendered jaw opening at speech onset, and lip sync restricted to low-energy consonants.

The decisive check is the channel-shift sweep. Speech content is held fixed while noise injected into the pauses is varied from none to strong. We plot idle jitter against noise level for the no-gate, VAD-freeze, augmentation and proposed models. The claim holds if the no-gate and augmentation curves rise on unseen noise types while the proposed curve stays flat, and if the proposed model has lower onset lag than VAD-freeze. The second check is a blind user study comparing the proposed model with VAD-freeze on naturalness during pauses. If users cannot tell the idle model from a frozen face, we drop the idle model from the design.