The goal of modern face synthesis has shifted from simply generating "realistic faces" to the precise, controllable manipulation of specific individuals. Whether it is dubbing a film into a new language, animating a static portrait, or swapping a user's likeness into a video, the core technical challenge remains the same: how do we disentangle a person’s unique identity from their transient expressions, poses, and lighting?
Researchers are increasingly moving away from holistic image generation toward modular frameworks that treat identity as a persistent "style" or "prior" that can be injected into various motion and structural templates.
Disentangling Identity from Attribute
To manipulate a face without losing the person's likeness, models must separate identity (the invariant geometry and texture of a person) from attributes (gaze, expression, and head pose).
Yufeng Zheng and colleagues demonstrated this with ST-ED (Self-Learning Transformations for Improving Gaze and Head Redirection), which uses an encoder-decoder structure to discover task-irrelevant factors like lighting and skin tone. By isolating these, the model can redirect a person's gaze in "in-the-wild" photos while keeping the person recognizable. Similarly, Daniel McDuff and the team behind DeepFN (DeepFN: Towards Generalizable Facial Action Unit Recognition with Deep Face Normalization) use self-supervised autoencoders to map diverse faces onto a common template, effectively stripping away identity to focus purely on facial expressions.
Grounding Pixels in 3D Structure
While 2D generative models like GANs and Diffusion produce stunning textures, they often struggle with spatial consistency during movement. To solve this, researchers are integrating 3D structural priors.
ShahRukh Athar and Dimitris Samaras introduced FLAME-in-NeRF (FLAME-in-NeRF : Neural control of Radiance Fields for Free View Face Animation), which combines the photorealistic rendering of Neural Radiance Fields (NeRF) with the explicit control of 3D Morphable Models (3DMM). By using a 3D mesh as a spatial prior, the system can disentangle facial expressions from the background, allowing for free-viewpoint animation from a single short video. This 3D-aware approach is echoed in MFIM (MFIM: Megapixel Facial Identity Manipulation) by Sanghyeon Na, which uses 3DMMs to supervise the training of StyleGAN generators, ensuring that identity remains stable even at megapixel resolutions.
The Role of Audio-Visual Synchronization
For applications like digital dubbing or talking avatars, the synthesis must be driven by audio. The breakthrough in this area came from the realization that "lip-sync" is a distinct skill that can be learned by an "expert" model.
K R Prajwal and Rudrabha Mukhopadhyay developed Wav2Lip (A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild), which uses a pre-trained, frozen discriminator to judge whether synthesized lip movements match the audio. This "expert" feedback forces the generator to produce near-perfect synchronization across any language or identity. More recently, Aggelina Chatziagapi and Hongyu Gong pushed this further with AV-Flow (AV-Flow: Transforming Text to Audio-Visual Human-like Interactions), using diffusion transformers to jointly synthesize speech, expressions, and head motion for 4D talking avatars.
From Global Tokens to Spatial Feature Maps
A major trend in identity preservation is the move from compressed identity "tokens" to high-resolution spatial maps. Traditional methods often compress a face into a single vector (like a CLIP embedding), which loses fine-grained details like scars, tattoos, or specific eye shapes.
FlashFace (FlashFace: Human Image Personalization with High-fidelity Identity Preservation), developed by Lianghua Huang and colleagues, avoids this compression. By using spatial feature maps and a disentangled integration strategy, it preserves the minute details of a reference photo even when the text prompt contradicts the image (e.g., putting a specific person in a costume they weren't wearing). Similarly, Face-Adapter (Face-Adapter for Pre-trained Diffusion Models with Fine-Grained ID and Attribute Control) by Yue Han allows for "plug-and-play" identity injection into pre-trained diffusion models without requiring expensive full-model fine-tuning.
The Ethical Imperative
As these technologies reach "megapixel" and "real-time" fidelity, the research community has reached a consensus: the ability to synthesize identity is inseparable from the risk of creating deceptive deepfakes.
Across nearly all recent work—from Yi Yang's multilingual dubbing (Large-scale multilingual audio visual dubbing) to Sanoojan Baliah's REFace (Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models)—there is a recurring call for integrated safeguards. The field is moving toward a dual-track development model where every leap in synthesis quality must be met with a corresponding advancement in digital watermarking, content verification, and detection frameworks.