Consistent Character Generation in Diffusion Models

166 concepts122 papers6 proposals

Modern text-to-image diffusion models excel at generating highly detailed, creative imagery from a single prompt. However, if you ask a model to generate "a hobbit walking through a forest," and then in a separate prompt ask for "a hobbit eating breakfast," you will almost certainly get two completely different characters. As noted by Jiahao Wang et al. in their work on OneActor, diffusion models are inherently stochastic and treat individual prompts as isolated events. They lack a persistent memory of specific subject details like facial features, hair texture, or clothing. This phenomenon is known as identity drift.

To unlock applications like storyboarding, graphic novel synthesis (Giannis Siglidis et al.), and long-form video generation (Weijia Wu et al.), researchers have developed techniques to enforce character identity consistency. Rather than relying on heavy per-subject fine-tuning, modern approaches modify the text embedding space, share attention layers across images, or leverage autoregressive history to keep characters recognizable across diverse scenes.


Why Identity Drift Happens

To understand the solution, we must understand why the problem exists. When a diffusion model processes a prompt like "a wizard," the text encoder (such as CLIP or T5) maps the word "wizard" to a high-dimensional text embedding. This embedding does not point to a specific person; instead, it represents a broad semantic region containing infinite possible wizards.

During the reverse diffusion process, the model starts with pure Gaussian noise. The denoising network uses cross-attention to steer the noise toward the "wizard" region. Because the initial noise is random, the model samples a completely different point within that semantic region every time. The specific facial structure, clothing patterns, and color palette are determined by this random seed.

To maintain consistency, we must force the model to land on the exact same sub-region of features across different prompts and random seeds.


Mechanism 1: Anchoring Identity in the Token Space

One intuitive approach is to narrow down the broad semantic region of a text prompt into a pinpoint coordinate that represents a single, unique individual. Instead of using generic words, can we give our character a unique "name" in the model's vocabulary?

Standard Prompt:  "A photo of a wizard cooking" ──> Broad semantic region (Identity drifts)
Anchored Prompt:  "A photo of [S*] cooking"    ──> Pinpoint coordinate (Identity preserved)

Baolu Li introduces CharacterFactory, which trains a lightweight GAN to map a latent space directly into the celebrity embedding space of CLIP. This allows the generation of consistent identities without requiring reference images. Similarly, Hsin-Ying Lee et al. propose CoCoIns, which maps a simple latent code to a "pseudo-word" representing a unique subject instance. By reusing this pseudo-word across different prompts (e.g., "A photo of [pseudo-word] reading a book"), the model is anchored to the exact same facial features.

Taking this further, Heliang Zheng et al. develop MagicNaming, which identifies a dedicated "Name Space" ($\mathcal{N}$ Space) within the existing text embedding architecture. By training an encoder to map any reference face into this Name Space, users can generate consistent fictional characters across diverse contexts as easily as they would for a well-known celebrity, bypassing the need for time-consuming inversion steps like those found in Dani Valevski et al.'s Face0.


Mechanism 2: Cross-Image Attention Sharing

What if we want to maintain consistency across a batch of images without altering the text embeddings? The answer lies in modifying how the model's attention layers operate during the generation process.

In a standard diffusion model, self-attention layers allow pixels within a single image to talk to each other to ensure internal structural coherence. However, different images in a batch are processed completely independently.

Standard Self-Attention:
Image 1 Latents ──> [ Self-Attention ] ──> Image 1 Output
Image 2 Latents ──> [ Self-Attention ] ──> Image 2 Output

Shared/Group Attention:
Image 1 Latents ──┐
                  ├──> [ Joint Attention Layer ] ──> Preserved Style & Identity
Image 2 Latents ──┘

Amir Hertz et al. introduce this concept with Style Aligned, a method that breaks image isolation by sharing attention across different generation tracks. By feeding the keys ($K$) and values ($V$) from a reference image's self-attention layers into the self-attention layers of target images, the target images are forced to share the same stylistic and structural properties.

This concept is generalized to unsupervised multitask learning by Lianghua Huang et al. in Group Diffusion Transformers (GDTs). GDTs concatenate attention tokens across an entire group of related images during generation. Because the attention mechanism can now look across image boundaries, the model inherently captures cross-image correlations, allowing it to preserve character identity and style across a sequence of frames without requiring explicit fine-tuning or external reference guidance.


Mechanism 3: Autoregressive History Conditioning

For long-form storytelling or multi-turn interactive dialogues (such as those explored by Minbin Huang et al. in DialogGen), generating all images simultaneously in a single batch is not always possible. Instead, the generation must be contextual and sequential.

Standard models treat each step in a dialogue or story as an isolated event, leading to the loss of the original subject's identity (Shichao Ma et al.). To solve this, autoregressive diffusion models condition the generation of the current frame on the multimodal history of previous frames and captions.

As demonstrated in Synthesizing Coherent Story with Auto-Regressive Latent Diffusion Models, injecting features from previous frames directly into the latent state or attention layers of the current frame prevents the model from drifting. Ying Shen et al. implement this in M2M (Many-to-Many) Diffusion, treating image generation as a contextual sequence where the model "perceives" prior images to generate new ones that maintain strict identity and style consistency.


Mechanism 4: Iterative Clustering and Self-Distillation

What if we want to create a brand new character from scratch using only a text prompt, but ensure they remain consistent across multiple future images? Without a reference photo, we face a chicken-and-egg problem.

Amir Hertz et al. solve this in The Chosen One via an automated pipeline of iterative clustering:

  1. Generate Candidates: The model generates a large batch of images from a single text prompt (e.g., "a detective"). Because of stochasticity, these images feature many different-looking people.
  2. Cluster: The system extracts visual features from these images and clusters them based on similarity, identifying the most cohesive group of images that happen to look like the same person.
  3. Distill: The model is fine-tuned on this distilled cluster, locking in that specific, stable identity for future generations.

A parallel strategy is proposed by Shengqu Cai et al. with Diffusion Self-Distillation (DSD). DSD leverages a self-distillation pipeline where the model generates and automatically curates its own synthetic training data, transforming an unsupervised consistency problem into a supervised optimization task that preserves identity across diverse scenes.


The Frontiers of Identity Preservation

As the field matures, the challenge has shifted from simple face preservation to maintaining holistic consistency across complex transformations. For instance, Sarah Mittenentzwei et al. highlight the difficulty of maintaining identity across different physiological states, such as aging or weight changes in medical storytelling.

In human animation and pose transfer, single-author works like Zhongcong Xu (MagicAnimate) and Armin Mustafa (UPGPT) demonstrate that integrating explicit structural priors (like DensePose or 3D SMPL models) is essential to prevent fine-grained details like clothing patterns from drifting during motion.

To track progress across these complex scenarios, benchmarks like MovieBench (Weijia Wu et al.), LoCoT2V-Bench (Chengyue Wu et al.), and IDEA-Bench (Chen Liang et al.) provide long-form, multi-scene evaluation suites. These benchmarks test whether models can maintain narrative logic and character continuity across changing camera angles, lighting conditions, and complex story arcs, pushing generative models closer to professional design standards.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_character_identity_consistency_34c4963f,
  title = {character identity consistency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/character-identity-consistency/rdr_34c4963f637c466798a4df7d91f71f59},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_character_identity_consistency_34c4963f_v1,
  title = {character identity consistency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/character-identity-consistency/rdr_34c4963f637c466798a4df7d91f71f59/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake