FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Cases

Shuai Tan, Bill Gong, Bin Ji, Ye Pan

TL;DR

FixTalk is an efficient, real-time talking head generation framework that eliminates identity leakage and rendering artifacts, even in extreme poses and expressions. By decoupling identity from motion features and cleverly recycling leaked identity data via a memory network, it delivers high-fidelity, decoupled facial control at over 27 frames per second.

What Problem Does This Paper Address?

Generating realistic talking head videos from a single source image and a driving audio/video track has massive potential for digital humans, filmmaking, and virtual assistants. To be practical, these systems need to be fast (efficient), allow independent control over different facial parts (decoupled control), and look realistic (high-quality rendering).

While diffusion models generate beautiful images, they are computationally heavy and slow. Generative Adversarial Networks (GANs) are much faster but struggle with two major flaws in "extreme cases"—such as sharp head turns or exaggerated emotional expressions:

  1. Identity Leakage (IL): The unique facial structure of the driving person accidentally bleeds into the source person's face.
  2. Rendering Artifacts (RA): Blurriness, distortion, or missing details occur because the model struggles to warp the face into extreme angles.
Figure 2. Illustration of identity leakage and rendering artifact.

FixTalk tackles both of these issues simultaneously without sacrificing real-time performance.

Methods

FixTalk builds upon a state-of-the-art GAN baseline (EDTalk) and introduces two lightweight, plug-and-play modules:

  • Enhanced Motion Indicator (EMI): The authors discovered that identity information is heavily coupled within the 4th layer feature map ($f^d_4$) of the driving motion encoder. To strip this identity data away, EMI introduces a lightweight cross-attention extractor with learnable queries (similar to a Q-Former) to isolate pure motion. It also applies a specialized disentanglement loss function to ensure the source and driving motion features remain independent.
  • Enhanced Detail Indicator (EDI): Instead of simply discarding the leaked identity information, the authors realized it could be useful. In self-driven scenarios (where the source and driving image are the same person), this "leakage" actually helps the model reconstruct missing details. EDI exploits this by using a dual-component Memory Network during training to store these detailed features. During cross-driven inference (different people), the model uses the source image and motion differences to query this memory, retrieving the exact high-frequency details needed to patch up rendering artifacts.

Datasets

The framework was trained and evaluated using several prominent talking head benchmarks:

  • VFHQ: Contains over 16,000 high-quality interview clips with a wide range of head poses, providing ample material to train the memory network.
  • MEAD: A large-scale audio-visual dataset featuring 60 actors displaying 8 different emotions, used to train and evaluate emotional expressiveness.
  • HDTF: An external high-resolution audio-visual dataset used strictly for testing to validate the model's generalization capabilities.

Key Insights and Ideas

  • Leakage is Layer-Specific: Identity leakage isn't a vague, distributed issue across the entire network. The authors pinpointed that it primarily originates from specific intermediate feature layers in the motion extraction process.
  • Taming a Bug into a Feature: The core "aha moment" of the paper is that identity leakage is actually a shortcut to high-quality rendering. In self-driven settings, leaking identity provides the network with the exact textures it needs for extreme angles. FixTalk successfully "tames" this phenomenon by storing it safely in an external memory bank during training, allowing the model to recall structural details during inference without corrupting the subject's identity.

Key Results

  • High Efficiency: FixTalk requires only 3.65 GB of GPU memory and runs at 27.6 FPS on an NVIDIA A100, comfortably clearing the 25 FPS hurdle required for real-time generation. This is a massive advantage over diffusion-based alternatives.
  • Superior Quality: Quantitative evaluations on the HDTF dataset show that FixTalk outperforms competitive GAN and diffusion baselines. It achieves a higher CSIM score (0.613), indicating superior identity preservation, alongside a lower NIQE score (13.44) and higher CPBD score (0.282), proving a significant reduction in artifacts.
Figure 1. Example animations generated by FixTalk. Given an identity source, FixTalk supports decoupled control over lip sync, head pose, and emotional expressions from their respective sources. As presented in Tab. 4, FixTalk achieves real-time generation. In particular, FixTalk demonstrates superior performance with high-quality rendering in extreme poses and expressive expressions.

Limitations and Future Work

Despite its strengths, the authors note a few limitations:

  • The model cannot automatically infer natural emotions or subtle micro-expressions directly from an audio track alone; it still relies on an explicit expression source.
  • While the mouth, pose, and expressions are successfully decoupled, eye movements remain linked to other facial features, which slightly limits total expressiveness. Future work will likely focus on fully isolating eye gaze control.

Summary updated

Abstract

Talking head generation is gaining significant importance across various domains, with a growing demand for high-quality rendering. However, existing methods often suffer from identity leakage (IL) and rendering artifacts (RA), particularly in extreme cases. Through an in-depth analysis of previous approaches, we identify two key insights: (1) IL arises from identity information embedded within motion features, and (2) this identity information can be leveraged to address RA. Building on these findings, this paper introduces FixTalk, a novel framework designed to simultaneously resolve both issues for high-quality talking head generation. Firstly, we propose an Enhanced Motion Indicator (EMI) to effectively decouple identity information from motion features, mitigating the impact of IL on generated talking heads. To address RA, we introduce an Enhanced Detail Indicator (EDI), which utilizes the leaked identity information to supplement missing details, thus fixing the artifacts. Extensive experiments demonstrate that FixTalk effectively mitigates IL and RA, achieving superior performance compared to state-of-the-art methods.
BibTeX
@inproceedings{tan2025fixtalk,
  author = {Shuai Tan and Bill Gong and Bin Ji and Ye Pan},
  title = {{FixTalk: Taming Identity Leakage for High-Quality Talking Head Generation in Extreme Cases}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure