When a talking-head model animates a single photo through a head turn toward full profile, it faces two kinds of pixels. Some pixels show surface that was visible in the source photo, such as the near cheek or the visible eye. Others show surface the model has never seen, such as the far side of the jaw or the inside of the ear. For visible surface, the right behavior is faithful copying. For unseen surface, faithful copying is impossible: any loss that punishes pixel error there pulls the output toward the average face, which is blurry. Current systems train every pixel with one fixed mix of pixel, perceptual, and adversarial losses. They also either ignore time or smooth it with optical flow, and optical flow breaks exactly where new surface comes into view. We propose to split the training loss by where each pixel's surface point was observed. Visible surface gets accuracy terms. Never-seen surface gets realism terms, plus a new temporal term that asks the model to stay consistent with its own earlier guesses about that surface. A frozen 3D head-correspondence network, used only at training time, tells us which is which.
The motivation comes from the perception-distortion tradeoff. The Perception-Distortion Tradeoff proves that no estimator can be both minimally wrong on average and statistically indistinguishable from real images, and that forcing full realism can double mean squared error. That bound depends on how much the input reveals. A head turn changes how much the source reveals, and the change happens per pixel and per frame. Our claim is that the tradeoff should be read locally. Then the fixed composite losses in MCNet, DaGAN, and THFEM are provably mis-weighted on both regions simultaneously. They over-constrain unseen surface, which causes blur at profile. They under-constrain nothing in particular, and their only temporal remedy discards the disoccluded pixels where flicker is worst.
Core Claim
The claim has three parts. First, the achievable perception-distortion curve is different for source-visible and never-seen surface. It also differs within a single frame, not just across yaw angles. Second, a composite loss that switches its terms by per-pixel observability reaches a better operating point on both curves than any single global weighting, including one tuned separately for each yaw bin. Third, temporal consistency for never-seen surface should be imposed as consistency with past generated frames, aligned through 3D canonical coordinates rather than 2D flow. Consistency with the driving video is the wrong target here, because the model's guess about the far cheek is not the true far cheek. The first two parts are also a measurement contribution, since nobody has reported the tradeoff stratified this way. The third part transplants an idea from video compression into a regime it was never designed for.
Relation to Prior Work
Three talking-head systems supply the backbone and the problem. MCNet and its scale-conditioned variant MCNet-scale fill missing anatomy during large turns by querying a global learned memory of facial patterns. They route half the channels to identity pass-through and half to memory refinement, under one fixed composite loss. The authors report that 90-degree turns remain hard and that slight flicker appears in long sequences. DaGAN adds self-supervised depth to stabilize rotations. FixTalk disentangles identity leakage from motion and reuses leaked details at extreme poses, but has no long-sequence temporal formulation. These papers motivate the proposal: each finds the profile failure. None changes what the loss asks for in unseen regions. We keep their architecture and change only the objective.
The temporal mechanism comes from compression. PLF-AR shows two things for video. A realism loss on the joint distribution of frames makes early errors permanent. A per-frame realism loss loses coherence. Judging the current frame jointly with past reconstructions avoids both problems. Its analysis covers planar motion on MovingMNIST and UVG patches, and it has no notion of self-occlusion. We borrow the conditioning-on-past-reconstructions idea. We then argue that error permanence is actually desirable for hallucinated surface, because the model should commit to its guess about the far cheek. It is harmful for visible surface, which must track the driver. Splitting by observability lets us choose permanence where we want it.
The observability signal comes from Densemarks. It maps head pixels, including hair and accessories, into a canonical 3D cube. It reports same-person matching error of 3.68 on CelebV-HQ, against 7.60 for DINOv3. Its roughly 16 GB memory cost would be prohibitive inside a generator. We only run it offline on training clips, so inference cost is unchanged.
Two lines of work constrain the design. Face restoration work, specifically VQFR and Panini-Net, shows that leaning on a generative prior yields sharp textures belonging to the wrong person. This rules out simply raising adversarial weight everywhere. It is also why visible surface keeps its accuracy and identity terms. KeyID handles identity drift by editing sparse clear keyframes and interpolating between them. That approach presupposes detectable frontal-ish faces, which is exactly what sustained profile removes.
Method
For each training clip, we run the frozen correspondence network on the source frame and every target frame. This gives each target pixel $p$ a canonical coordinate $u_t(p)$.
A target pixel is source-visible if some source pixel has a canonical coordinate within radius $r$ of $u_t(p)$. It is previously generated if a generated frame within the last $k$ steps covered that coordinate. Otherwise it is new. The soft masks $m^{src}_t$ and $m^{prev}_t$ record the first two conditions. The model is trained in the usual self-reenactment setup: source and driver come from the same video, so ground truth exists for every region, including the ones hidden in the source.
The objective for generated frame $\hat{x}_t$ with ground truth $x_t$ is
$$ \mathcal{L}_t = \big\| m^{src}_t \odot (\hat{x}_t - x_t) \big\|_1 + \lambda_{perc}\,\mathcal{L}_{perc}\big(m^{src}_t;\hat{x}_t,x_t\big) + \lambda_{id}\,\mathcal{L}_{id} + \lambda_{adv}\,\mathcal{L}_{adv}\big(\hat{x}_t \mid \tilde{x}_{t-k:t-1}\big) + \lambda_{T}\,\big\| m^{prev}_t \odot (1-m^{src}_t) \odot (\hat{x}_t - \tilde{x}_{t-1}) \big\|_1 . $$
Here $\odot$ is per-pixel multiplication, and $\mathcal{L}_{perc}$ is a VGG feature distance restricted to the masked region. $\mathcal{L}_{id}$ is the backbone's existing identity term. $\tilde{x}_{t-1}$ is the previous generated frame, re-sampled into frame $t$ through canonical coordinates rather than optical flow. Where a canonical point is visible in both frames, its pixel is carried along. The adversarial term uses a discriminator that sees the current frame together with canonically re-sampled past generated frames. Its real examples are real frames paired with re-sampled past real frames, in the style of PLF-AR.
Read in words, the objective does three things. Visible surface is held to the truth. Unseen surface is held only to looking real given what the model already drew. The temporal term applies only where the model is guessing, so it never fights the driver on visible surface. We also train variants that keep a small L1 weight $\epsilon$ on unseen pixels. Sweeping $\epsilon$ and $\lambda_{adv}$ traces the tradeoff curve for each region separately.
Research Program
The work proceeds in three stages.
Stage 1: measurement. We take the released-architecture MCNet, trained with its own losses, and sweep the adversarial weight. We report distortion (PSNR, LPIPS) and realism (FID on masked crops) separately for source-visible and new pixels, in yaw bins of 0–15°, 15–45°, 45–70°, and 70–90°. This establishes whether the two regions really lie on different curves. If the curves coincide, the per-pixel idea collapses into per-yaw weighting and we would report that.
Stage 2: accuracy-realism split. We retrain MCNet with the visibility-split loss, without the temporal term, and compare the resulting curves.
Stage 3: temporal term. We add the canonical-space temporal term and the history-conditioned discriminator. We then study error permanence directly.
Baselines
The naive reference point is a plain frame-by-frame generator on the same backbone trained with L1 alone. It shows how much blur the averaging effect produces at profile, and it anchors the distortion axis.
The strongest existing method is MCNet (MCNet). Its memory module is the most direct existing answer to missing anatomy under large turns. On VoxCeleb1 it reports SSIM of 82.5%, compared with 72.3% for FOMM and 81.6% for TPSM. We also report DaGAN's published VoxCeleb1 numbers (DaGAN) as a reference: CSIM 0.899, PSNR 31.220, head-pose error 1.22. For fairness, every variant, including ours, uses MCNet's architecture, VoxCeleb1 training split, iteration count, and resolution. Each variant gets the same hyperparameter-search budget over its loss weights, so the proposal cannot win by extra tuning. Beating MCNet means lower LPIPS and FID on new pixels in the 45–90° bins, without losing more than 0.3 dB PSNR on source-visible pixels. It also means lower canonical-space flicker on disoccluded regions, and no drop in whole-frame SSIM relative to the 82.5% level measured under our reproduction. We report FixTalk (FixTalk) qualitatively where code allows. The file contains no reported HDTF metric values for FixTalk, only 3.65 GB memory and 27.6 FPS.
Three ablations isolate the key ingredients. The first is a per-frame yaw-conditioned scalar: the same loss terms, but with weights set by a function of head yaw instead of per-pixel masks, given an equal tuning budget. It tests whether per-pixel observability matters beyond pose. The second replaces canonical re-sampling with occlusion-masked optical-flow warping, testing whether 3D correspondence matters for time. The third conditions the discriminator on past real frames rather than past generated frames, testing the PLF-AR ingredient.
Why not just…?
Why not just tune the loss weights as a function of yaw? This is the most serious threat, and the file's own framing of the gap is angle-based. The argument against it is that at 30–60° yaw, a single frame contains both a well-observed near half and an unobserved far half. Any one scalar must be wrong for one of them. That argument could fail if the network learns internally to route realism to the far side. The yaw-scalar ablation, run with equal tuning budget and stratified by mask, settles the question. If it matches us within noise at mid yaw, we would drop the per-pixel claim and report the per-yaw schedule as the finding.
Why not just add a standard flow-warped temporal loss with occlusion masking? Occlusion masking removes disoccluded pixels from the temporal loss entirely. It therefore leaves unconstrained exactly the pixels where hallucination flickers. Without masking, flow gives meaningless correspondences across silhouette changes, as DaGAN and MCNet observe in boundary tearing under large rotations. The flow-versus-canonical ablation measures flicker in the disoccluded region specifically. A tie there would mean 3D correspondence is unnecessary.
Why not just raise adversarial weight, or feed the previous frame autoregressively? Raising realism everywhere buys sharpness at the cost of identity, as VQFR and Panini-Net document for face restoration. The Stage 1 sweep will show this as rising error on source-visible pixels. Naive autoregression makes every error permanent, as PLF-AR proves. Our visibility split confines permanence to guessed surface. The artifact-injection check below tests that directly.
Decisive Experiments
Training uses VoxCeleb1. Evaluation covers VoxCeleb1, HDTF, and CelebV-HQ clips binned by estimated yaw. Two datasets give true novel-view ground truth. On MEAD's synchronized multi-camera recordings, we animate from the frontal camera and compare against the real side camera at the same timestamp. On Multi-PIE's discrete angles up to 90°, we do the same for static poses. Identity at profile is measured against a real profile image of the same person, not the frontal source, because face recognizers degrade across large pose gaps. For MEAD, OPT's reported CSIM of 0.88 (OPT) is a frontal-regime reference, not a profile one.
Three results decide the proposal. The first is whether Stage 1 shows separate tradeoff curves for visible and new pixels. The second is whether the visibility-split loss beats the yaw-scalar ablation in the 30–60° bins on new-pixel LPIPS and FID at matched visible-pixel PSNR. The third is an artifact-injection test. We corrupt one generated frame in a visible region and in a hallucinated region, then measure how many frames the corruption persists. The prediction is fast correction on visible surface and stable, non-flickering persistence on hallucinated surface. Flow-based and naive autoregressive baselines should show either persistence everywhere or flicker everywhere. If the third prediction fails, the temporal claim is wrong even if the loss split helps.
References
- The Perception-Distortion Tradeoff
- Implicit Identity Representation Conditioned Memory Compensation Network for Talking Head Video Generation
- Depth-Aware Generative Adversarial Network for Talking Head Video Generation
- Exploring Talking Head Models with Adjacent Frame Prior for Speech-Preserving Facial Expression Manipulation
- Learning Implicit Scale Conditioned Memory Compensation for Talking Head Generation
- FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Cases
- Perception Loss Function Adaptive to Rate for Learned Video Compression
- Densemarks: Learning Canonical Embeddings for Human Heads Images via Point Tracks
- VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder
- Panini-Net: GAN Prior Based Degradation-Aware Feature Interpolation for Face Restoration
- KeyID: Decoupled Drafting and Keyframe Editing for Identity-Preserving Video Generation
- OPT: ONE-SHOT POSE-CONTROLLABLE TALKING HEAD GENERATION