Audio-Timed Gates on Skip Connections for High-Detail Lip-Sync Video

Talking-face generators face a tug-of-war. To keep fine detail such as skin pores, teeth and hair, they feed a sharp reference photo of the person into the generator through high-resolution side paths. Those side paths are so informative that they drown out the audio. The mouth then "freezes" into the reference photo's pose instead of following speech. We propose to put a small learned gate on each skip connection of a frozen video diffusion model. A skip connection is the shortcut that carries full-resolution encoder features straight to the decoder. A video diffusion model is a generator that produces video by gradually denoising random noise. The gate's opening is driven by the audio at that moment, so reference detail flows freely where and when the speech does not demand change. Where the audio calls for a new mouth shape, the gate yields to audio-driven content. Each skip level reads the audio over a window matched to that level's temporal reach: short windows for fine-texture levels, longer windows for coarse layout levels.

The bet is easy to falsify. Existing fixes for identity freezing are static: they drop conditions during training, apply a fixed mouth mask, or set a global guidance strength. Each one slides a system along a fixed trade-off curve between lip-sync accuracy and detail fidelity. We claim that a gate which varies in time with the speech moves the curve itself. If a carefully tuned static mask reaches the same curve, the idea is wrong, and we will say so.

Core Claim

SayAnything diagnosed the problem we target. It runs audio-driven lip sync inside Stable Video Diffusion and observes that rich reference features, such as those from a ReferenceNet (an auxiliary copy of the denoising network that encodes the reference image), overpower audio and lock the mouth. Its remedy is to remove the heavy reference branch, replacing it with a 98M-parameter convolutional identity encoder, and to drop the reference condition whenever audio is dropped during training. MAGIC-Talk takes the other side of the trade-off. It keeps a full ReferenceNet for identity and moves motion into a separate network driven by 3D face-model and edge proxies.

Both papers motivate this proposal, and both treat the reference-versus-audio balance as a property fixed at training time. Neither lets the balance change frame by frame. SCEdit supplies our method. It shows that skip connections in a diffusion U-Net carry most fine structure, and that small bottleneck "tuners" inserted on those skips can steer generation while the encoder stays frozen. SCEdit works only on still images and has no temporal or audio input. Our contribution is to make the skip tuner a time-varying, audio-controlled gate inside a video model.

Concretely, let $h_{l,t}$ be the skip feature at U-Net level $l$ and frame $t$. Let $a_{l,t}$ be the audio features resampled to the video frame rate and pooled over a window whose length equals the temporal receptive field of level $l$. The tuned skip is

$$\tilde h_{l,t} = (1-g_{l,t})\odot h_{l,t} + g_{l,t}\odot \phi_l(h_{l,t}, a_{l,t}),$$

where $\phi_l$ is a small SCEdit-style bottleneck that writes audio-driven content and $g_{l,t}\in[0,1]^{H_l\times W_l}$ is a spatial gate predicted from $a_{l,t}$ and $h_{l,t}$. Here $H_l\times W_l$ is the spatial size of level $l$ and $\odot$ is element-wise multiplication. When $g$ is near zero the skip passes reference-rich detail untouched. When $g$ is near one, audio-driven content replaces it at that location.

Matching window length to level borrows the mirrored-receptive-field idea from Multi-Temporal Lip-Audio Memory. That paper aligns audio and lip features layer by layer for lip reading, which is recognition, not generation. We test whether the same matching helps when the target is synthesis.

Research Program

We build on a frozen Stable Video Diffusion backbone, the same family SayAnything uses. Audio enters through Whisper features injected by cross-attention, as in that paper, so that the only architectural difference is the gated skip tuners. We train only the tuners and the audio cross-attention.

The work has three parts. The first is the gated tuner itself, with gates initialized closed so that training starts from the reference-preserving solution.

The second is a set of diagnostics showing that the gate learns timing rather than a disguised fixed mask. We will measure how strongly the gate's opening correlates over time with ground-truth mouth aperture. We will shift the audio by $k$ frames and check that the gate's response shifts with it. We will also feed silence and check that the gates close and the output collapses to the reference.

The third is a controlled study of the scale-to-window assignment. We compare matched windows against a single shared window and against a deliberately inverted assignment, with long windows at fine levels and short windows at coarse ones.

We make no latency claim. The tuners do not reduce the number of denoising steps, so SayAnything's roughly 7 seconds per second of video on an RTX 4090 is not addressed here. Training memory should fall relative to full fine-tuning, as SCEdit found for images, and we will report it.

Baselines

The naive reference point is the frozen backbone with audio cross-attention and the reference image injected, and no skip intervention. It should show identity freezing at its worst and sets the floor on lip-sync accuracy.

The strongest existing method is SayAnything, which reports FID 5.68 and FVD 197.71 on HDTF. We cite that published number. Because it was trained on about 600 hours drawn from AVASpeech, VFHQ and MultiTalk, a fair comparison also requires reimplementing its ID-Guider and synchronized dropout on our backbone, data split, step count and hyperparameter search budget. Beating it means improving lip sync at equal or better FID/FVD and identity similarity on the reimplementation, with the published figure shown alongside.

The most important ablation is a static gate. It uses the same tuners, but the gate is a learned map that does not depend on audio and is constant across frames: in effect, a learned mouth mask. A second ablation sweeps a hand-drawn rectangular mouth mask together with the audio guidance scale. That gives each static method its own curve of lip sync versus detail, and our claim requires the audio-timed gate's curve to lie strictly beyond those curves. A third ablation removes the gate and keeps only the audio-conditioned tuner residual. It tests whether gating matters, or whether merely adding audio to the skips does the work.

As non-diffusion reference points on HDTF, we include Flow-Guided One-Shot, which warps pixels from the source image using 3D face-model flow and reports PSNR 24.41 dB and SSIM 0.84. We also include LawDNet, which warps features with learned local affine transforms. For LawDNet the file has only relative gains (a 3.07% SSIM increase and a 6.33% LSE-D improvement), so no absolute HDTF numbers were found on Lacuna. These two represent the "borrow texture by warping" family.

Why not just…?

Why not just tune SayAnything's condition dropout and audio guidance strength? This is the closest competitor, and it may be enough. Guidance strength is a single global scalar applied equally to every frame and every location. It can trade detail for motion, but it cannot keep teeth sharp in a closed-mouth frame while freeing the lips in the next one. The guidance sweep baseline settles the question directly. If its curve matches ours, the gate adds nothing.

Why not just mask the reference out of the mouth region? This is the real threat, since a fixed mask already separates "keep detail" from "follow audio" in space. Its weakness is that the mouth interior, especially the teeth, needs reference detail when the mouth is open and needs none when it is shut, and that requirement changes with each phoneme. Both the static-gate and hand-mask ablations exist to test this point. We will report honestly if they close the gap, and the gate-timing diagnostics will show whether our gate is behaving as a mask anyway.

Why not just generate at low resolution and super-resolve? Flow-Guided One-Shot argues that low-resolution bottlenecks discard detail that later upsampling cannot recover. Super-resolution applied frame by frame or window by window also introduces flicker and teeth morphing. We will run a 256-pixel model followed by a generic upsampler as an extra point on the curve and measure temporal flicker with FVD.

Why not just fine-tune the whole backbone? SCEdit reports that skip tuning reaches FID 13.82 against 15.48 for base SD v1.5 on COCO2017, with 52% less memory. Full fine-tuning on talking heads also risks erasing the backbone's general visual knowledge. We will report one full fine-tune run, at matched wall-clock budget, as a reference.

Proposed Experiments

We train on HDTF and evaluate on its held-out subjects at 512 resolution. We also evaluate zero-shot on LRS2 and CMLR. On those sets STSA, a dual-path warping method for visual dubbing, reports FID 24.23 and PSNR 31.73 dB on LRS2, and FID 17.07 and PSNR 30.21 dB on CMLR. STSA warps reference features along a spatial path and a temporal path and forces the two to agree, so it is a contrasting design that also keeps detail without diffusion. It was trained in-domain on LRS2, so on that set we use it as a reference rather than a like-for-like competitor.

Metrics fall into four groups. For lip sync we use LSE-D and LSE-C. Because SayAnything warns that SyncNet rewards stiff motion, we add an AV-HuBERT-based synchronization score in the spirit of AV-Lip-Sync+. We also measure the ratio of generated to ground-truth variance in mouth aperture, which directly quantifies stiffness. For detail we use FID, FVD, CPBD sharpness and high-frequency energy computed on the mouth and teeth crop. For identity we use face-embedding cosine similarity.

The decisive check is the curve of lip sync against detail. Each static baseline is swept over its knob, and the audio-timed gate must lie beyond all static curves on HDTF and in at least one zero-shot set. Two secondary checks complete the evaluation. Under the audio-shift test, the gate's response must follow the shifted audio. In the scale study, the matched-window assignment must beat the inverted assignment, and if it does not, the multi-scale part of the claim is dropped.