Sharpen Only What the Audio Determines: Information-Guided Decoding for Talking-Face Synthesis

Speech-driven face generators can now fool people. In one study, viewers guessed at chance whether a video was real or synthetic. Yet the same synthetic faces helped listeners understand speech in noise far less than real video did: 62% word comprehension, against 71% for natural video and 51% for audio alone (Speech-Driven Facial Animations Improve Speech-in-Noise Comprehension of Humans). The faces looked right but carried less of the speech. We read this as a precise failure of how realism is usually enforced. Realism checks ask whether each frame looks like a real face. They do not ask whether the face is the right face for this audio. A generator can therefore pass them while smoothing away inner-lip closures and teeth visibility, which are exactly the cues that carry phonetic information.

We propose to fix this at decoding time inside a coarse-to-fine image generator. The generator is a next-scale autoregressive model, which produces an image as a pyramid of discrete token maps, from a 1×1 summary up to a full-resolution grid, each scale predicted from the coarser ones. For every token, we measure how much the audio changes the model's predicted distribution. Tokens the audio strongly determines are decoded sharply, close to the most likely value. Tokens the audio does not determine, such as skin texture, hair, blinks, and much of head motion, are sampled freely so the output stays realistic. The two-sentence version is simple. Treat each piece of the face as deterministic or stochastic according to how much the driving signal pins it down, not according to its spatial scale. We expect this to recover articulatory information without giving up realism.

Core Claim

The perception-distortion tradeoff (The Perception-Distortion Tradeoff) has two parts. Distortion measures how far the output is from the true image. Perception measures how far the distribution of outputs is from the distribution of real images. The standard result is that minimizing average error yields the blurry conditional mean, and that forcing perfect realism raises the minimum squared error by at most a factor of two (Rethinking Lossy Compression: The Rate-Distortion-Perception Tradeoff).

That cost is not spread evenly. Where the posterior given the inputs is narrow, sampling and averaging nearly coincide, so realism is almost free. Where the posterior is broad, the mean is blurry and realism must be paid for in error.

Hierarchical restoration methods convert this insight into a frequency rule: a deterministic network handles low frequencies and a stochastic prior handles high-frequency texture. Learning Diffusion Texture Priors for Image Restoration (DTPM) does exactly this, and coarse-to-fine generators implicitly assume the same split. Our claim is that for talking faces this rule is wrong in both directions:

  • Audio-determined detail can be fine-scale. The cues that carry phonetic information, such as lip-seal edges, the teeth boundary, and labiodental contact, are small, high-frequency, and strongly determined by the audio.
  • Unconstrained structure can be coarse-scale. Large, low-frequency structure like head pose, gaze, and eyebrow motion is barely constrained by audio.

The correct partition is therefore by conditional determinacy, which can be read directly from the generator itself. Our prediction is that per-token, information-guided decoding traces a three-way frontier that dominates every globally tuned setting. The three axes are realism (FID/FVD), geometric fidelity (landmark distance), and articulatory information (lipreading accuracy). In particular, it should dominate the "coarse sharp, fine stochastic" schedule.

Research Program

Base model. We build a talking-face generator on next-scale prediction (Visual Autoregressive Modeling). VAR reports an ImageNet 256×256 FID of 1.73 and roughly 20× faster sampling than raster-order autoregression.

  • Each frame is tokenized into a multi-scale pyramid.
  • A transformer predicts the scales of frame $t$ conditioned on the identity reference image, the previous frame's tokens, and a window of audio features.
  • Conditioning is injected the way ControlVAR injects spatial controls scale by scale, extended with a temporal and audio stream that ControlVAR lacks.

Like Taming Transformer for Emotion-Controllable Talking Face Generation, we use discrete visual tokens and an emotion input. Unlike it, we predict scale by scale rather than as a flat token sequence. The model is trained with audio dropout so that both audio-conditioned and audio-free predictions are available, as in standard classifier-free guidance.

Measuring audio information per token. For token $i$ with context $c_i$ (identity, coarser scales, previous frame) and audio $a$, we define the audio information gain as

$$g_i = \mathrm{KL}\big(p(z_i \mid c_i, a)\,\|\,p(z_i \mid c_i)\big),$$

where $z_i$ is the token's codebook index. The quantity $g_i$ is large when hearing the audio substantially reshapes what the token should be. It costs nothing extra, because both distributions are already computed for guidance.

Decoding rule. Each token is decoded at temperature $\tau_i = \tau_0 e^{-\beta g_i}$ with guidance weight $w_i = w_0 + \gamma g_i$. Here $\tau_0$ and $w_0$ are base values and $\beta,\gamma \ge 0$ control how strongly information gain sharpens decoding. With $\beta=\gamma=0$ the rule reduces to ordinary uniform sampling. Large $\beta$ drives audio-determined tokens toward the mode while leaving audio-independent tokens fully stochastic.

The same construction applies to the identity reference. Sharpening tokens with high identity information gain is our test for whether identity drift under extreme expressions can be reduced selectively. That drift is reported on MEAD, and identity hallucination by generative priors is documented in Panini-Net and VQFR.

Contrast with prior variance-control methods. This differs from Pref-Restore in two ways. Pref-Restore cuts output variance across random seeds by about half for every pixel of a static face, using preference reinforcement learning. We cut variance only where a condition determines the answer, and without retraining. It also differs from Cafe-Talk's mask-based guidance (Cafe-Talk), which uses hand-set time masks to bound when an action unit applies. Our masks are per token, computed by the model, and they change when the set of conditions changes.

Analysis. Before any optimization, we map $g_i$ across scales, face regions, phoneme classes, and emotions. This map tests the central factual premise: that audio information concentrates at mid-to-fine scales around the mouth and is low for coarse head pose. If instead it tracks scale monotonically, the frequency-split rule is already adequate, and the contribution shrinks to the base architecture.

Baselines

Naive reference. Our own model with uniform sampling: one global temperature and guidance weight, swept over a grid. This sweep is the global perception-distortion frontier that loss or temperature tuning can reach. Our method must lie strictly beyond it, improving lipreading accuracy at matched FVD or FVD at matched lipreading accuracy. A gain within seed-to-seed spread does not count.

Scale-schedule ablation. The same model with temperature set by scale alone: coarse scales near-greedy, fine scales sampled, with the crossover scale tuned. This is the DTPM-style structure-versus-texture split ported into the pyramid, and it isolates our key ingredient. Beating it at matched tuning budget, defined as the same number of evaluated configurations on a held-out validation split, is the core result.

Fixed-mouth-mask ablation. Greedy decoding inside a fixed lower-face mask and sampling elsewhere. This tests whether a hand-drawn region does the job of the information map.

Strongest end-to-end pixel competitors. AvatarSync is the closest hierarchical talking-head system: autoregressive keyframes followed by learned interpolation. It reports FID 23.69 and FVD 251.89 on HDTF. On CREMA-D, Emotionally Enhanced Talking Face Generation reports FID 6.04, emotion accuracy 83.20%, and lip-sync error distance 7.17. For expressive control on MEAD, DreamTalk reports 86.7% style accuracy.

Motion-space competitor. Ditto reports the lowest FID/FVD on HDTF100 and a real-time factor below 1, but no reported result was found on Lacuna for its exact FID/FVD values.

Fairness protocol. Where code is available, we retrain or evaluate each competitor on our splits with identical face crops, resolution, and metric implementations. Published numbers serve only as sanity anchors, because their protocols differ. Matching compute means comparable training GPU-hours for our base model, and all decoding variants share one trained network.

Why not just…?

Why not just tune the loss weights or a global temperature? Doing so slides the whole output along a single frontier. The perception-distortion result (Blau and Michaeli) says such a frontier exists but does not say it is reachable by one knob. Varano et al.'s generator, trained with balanced reconstruction and adversarial terms, was perfectly realistic yet lost a third of natural video's comprehension benefit. The global sweep is our naive baseline. If it matches our frontier, the proposal fails.

Why not just generate motion in a compact space and render it, as Ditto or DreamTalk do? This is the most serious threat. A low-dimensional motion space, made of keypoints or face-model parameters, is nearly deterministic given audio and sidesteps pixel hallucination. Our bet is that such spaces under-represent the inner mouth (teeth, tongue, inner-lip seal), which is where Varano et al. located the missing cues, while the renderer still samples texture blindly. The test is mouth-region lipreading accuracy on GRID and a matched-realism comparison against the motion-space baseline. If the motion-space pipeline equals us on lipreading at equal FVD, we will report that the pixel-level allocation is unnecessary when a good motion space exists.

Why not just use a fixed mouth mask, or the DTPM-style coarse-sharp/fine-stochastic split? Both are cheaper and need no information map. The case against them is that determinacy depends on which conditions are supplied and on the phoneme. When an emotion label or action-unit control is given, brow and cheek tokens become determined, and a fixed mask cannot follow that shift. During silences, the mouth should be sampled rather than frozen. Both appear as ablations above. If the mask ties our method, the honest conclusion is that a simple region heuristic suffices for audio, and the information map's value reduces to multi-condition settings, which we test separately.

Proposed Experiments

Datasets and metrics. We train on HDTF and MEAD and evaluate on HDTF100, MEAD (emotion), CREMA-D, and GRID. GRID's small vocabulary makes automatic lipreading a reliable measure of articulatory information. The metrics are:

  • realism: FID and FVD;
  • geometric and audio-visual fidelity: landmark distance, SyncNet confidence/distance, and ArcFace identity similarity to the reference;
  • emotion: emotion accuracy;
  • articulatory information: word accuracy of a fixed audiovisual recognizer run on muted video;
  • conditional realism: FVD computed on concatenated audio-visual features, which captures the joint distribution that frame-wise realism ignores.

Decisive checks. There are four.

  1. Information map. Does $g_i$ concentrate at mid and fine mouth scales during consonant closures, and is it low for coarse pose? This verifies or kills the premise.
  2. Frontier dominance. On HDTF and GRID, does information-guided decoding dominate the global sweep, the scale schedule, and the fixed mask on the realism-versus-lipreading plane at matched tuning budget?
  3. Adaptation to conditions. On MEAD, does adding emotion as a second information-gain channel improve emotion accuracy without raising sync error? And does selective identity sharpening improve ArcFace similarity on strong-intensity clips relative to uniform sharpening, which should instead hurt FID?
  4. Human comprehension. A small speech-in-noise study on GRID asks what fraction of the gap between Varano et al.'s 62% synthetic and 71% natural comprehension the method closes, with a Turing-style realism check run alongside.

The project succeeds if checks 1 and 2 hold. It is informative even if check 2 fails against the motion-space or mask baselines, because the information map itself would settle where speech actually lives in a face's generative hierarchy.