Identity Preservation in Generative Image Synthesis

21 concepts20 papers1 proposal

When modern generative models are asked to modify an image—whether by changing a person's pose, adding an expression, simulating aging, or filling in an occluded region—they face a fundamental tension between plausibility and identity.

Deep generative networks excel at creating highly realistic human features because they map inputs to a learned manifold of "all possible faces." However, when a model is tasked with editing a specific individual, this strength becomes a liability. Left unconstrained, the model defaults to the most statistically probable features in its training data. Instead of restoring the unique shape of a subject's eye or the specific texture of their skin, it hallucinates a generic, average substitute. This phenomenon is known as identity drift.

To solve this, a major research direction focuses on subject-specific feature retention. The core intuition is simple: we must anchor the generative model's latent space using explicit personal priors, ensuring that while attributes like pose, age, or environment change, the underlying identity remains invariant.


Anchoring Generation with Exemplar Priors

The earliest and most intuitive way to prevent identity drift is to feed the generative network an explicit cheat sheet: a reference image (or "exemplar") of the target subject.

In standard image completion, a network evaluates the surrounding pixels to guess what belongs in a missing region. If a person's eyes are closed or blocked, the network draws a plausible pair of generic eyes. To bypass this, Brian Dolhansky et al. introduced Exemplar GANs (ExGANs). By passing a reference photo or a compressed perceptual code of the subject directly into the generative process, the network is forced to reconstruct person-specific details—such as a precise iris color or ocular shape—rather than inventing them from scratch.

Standard Inpainting:  Context Pixels ───> [ Generative Model ] ───> Generic Face
Exemplar Inpainting:  Context Pixels ───┐
                      Reference Photo ──┴─> [ Generative Model ] ───> Identity-Consistent Face

This reference-driven anchoring is highly effective for overcoming physical obstructions. For instance, Virtual Reality headsets completely block a user's upper face during telepresence. Yajie Zhao et al. demonstrated that utilizing an external reference image allows an inpainting framework to reconstruct the occluded ocular region accurately across various head poses. Similarly, Zheng Chen et al. showed that incorporating a static RGB reference image as a prior allows models to preserve fine traits like skin tone when interpreting partial infrared observations from a headset, proving far more reliable than pure generative hallucination.


Attention-Driven Feature Routing

While providing a reference image helps, the next challenge is architectural: how do we ensure the model actually pays attention to the identity details without destroying its ability to perform the requested edit? If you simply concatenate the reference image to the input, the network often overrides the fine-grained details during deep convolutions or diffusion timesteps.

Modern frameworks solve this by using parallel attention pathways to isolate and inject identity features.

The authors of Personalized Face Inpainting with Diffusion Models by Parallel Visual Attention developed Parallel Visual Attention (PVA). Instead of forcing identity information through the main text-to-image generation pipeline, PVA uses a lightweight, parallel pathway that injects reference features directly into a frozen diffusion model's attention layers. This allows users to perform complex language-based edits (e.g., "add a beard") while keeping the core facial structure untouched.

A similar strategy is employed by the authors of PATMAT, who fine-tune a mask-aware transformer by using a small set of reference images to create stable "anchors" within the transformer's style space. This targeted tuning ensures the model retains unique facial textures and brow structures during restoration.

When editing complex scenes containing multiple people, identity retention becomes even more difficult because the model can accidentally swap features between individuals. To address this, Nanxuan Zhao et al. introduced a "Person-Aware Attention" module for group portrait editing. This architecture explicitly maps specific reference features to corresponding individuals in the image, allowing the model to generate realistic physical interactions between people while preventing their identities from bleeding into one another.


Disentangling Identity from Attributes

When a model performs a macro-transformation on a face—such as modifying an expression, altering a pose, or simulating decades of aging—the identity features and the attribute features often become entangled. If the model doesn't explicitly understand the difference between "the structural shape of this person's jaw" and "the wrinkles caused by aging," it will inadvertently alter both.

To achieve true feature retention, models must explicitly factorize the latent space into orthogonal components: one for identity (which stays constant) and one for the transformation (which varies).

                      ┌───> Identity Vector (Constant) ───┐
Input Image ───> [Decomposition]                          ├───> [Synthesis] ───> Aged Face
                      └───> Age Vector      (Modified) ───┘

Zhizhong Huang et al. implemented this concept through a multi-task learning framework called MTLFace. By using an attention-based decomposition mechanism, the network splits an image into distinct identity and age representations. Because the identity vector is completely isolated, the model can manipulate the age vector to smoothly transition a face from youth to old age without altering the underlying facial signature. This ensures that the generated images remain recognizable to downstream face recognition systems, a concept the authors further validated across large-scale cross-age benchmarks in subsequent work.


Grounding with 3D Structural Priors

Purely 2D generative models lack an inherent understanding of physical geometry, making them prone to identity drift when a subject is rotated or viewed from a sharp angle. To maintain identity across extreme spatial transformations, researchers ground 2D generative models using explicit 3D structural priors.

By anchoring the pixels to a rigid or parametric 3D shape, the model is constrained by physics. Kripasindhu Sarkar et al. tackled the problem of re-posing humans from a single image by combining a parametric 3D body model with high-dimensional neural textures (UV feature maps). The 3D model handles the geometric rotation, while the neural textures preserve the intricate, identity-specific clothing and facial details, preventing the subject's appearance from shifting as they turn.

This synergy between 3D structure and generative priors has become central to modern avatar creation. Zichen Tang et al. introduced GaussianIP, a framework that pairs 3D Gaussian Splatting with a human-centric diffusion prior. While standard text-to-3D models often generate generic human figures, GaussianIP uses image-based portrait conditions to guide the 3D generation process. The 3D Gaussians anchor the physical structure in space, while the diffusion prior ensures that the fine-grained facial details remain perfectly consistent and recognizable from any viewing angle.

Key Foundational Work

Learning Face Age Progression: A Pyramid Architecture of GANs (2018) — contributed 1 concept elements to this direction.
Eye In-Painting with Exemplar Generative Adversarial Networks (2018) — contributed 1 concept elements to this direction.
You Said That?: Synthesising Talking Faces from Audio (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_subject_specific_feature_retention_e71d557b,
  title = {subject-specific feature retention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subject-specific-feature-retention/rdr_e71d557b14054850be945cd52219982e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_subject_specific_feature_retention_e71d557b_v1,
  title = {subject-specific feature retention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/subject-specific-feature-retention/rdr_e71d557b14054850be945cd52219982e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake