Preserving Temporal Identity Consistency in Video Generation

444 concepts341 papers3 proposals

When humans watch a video, we take for granted that an object's core features—the specific texture of a jacket, the unique geometry of a face, or the pattern on a dog's fur—remain stable from one frame to the next. However, for generative video models, maintaining this stability is a profound challenge.

As a generated video progresses, subjects frequently undergo identity drift (or appearance drift), gradually morphing into generic versions of themselves or losing fine-grained details entirely. Understanding why this happens, and how modern architectures fix it, requires looking at the fundamental tension between motion and memory in diffusion models.

The Root Cause: The Autoregressive Trap

To understand why identity drift occurs, imagine drawing a flipbook animation. If you create each new page by looking only at the page you just finished, any tiny error—a slightly elongated nose or a misplaced line—will be carried over and amplified on the next page. By the time you reach page 50, your character will look completely unrecognizable.

This is the autoregressive error accumulation problem. As noted in a comprehensive review by Wenping Ma 0001 et al. in Video diffusion generation: comprehensive review and open problems, models frequently struggle to balance physically plausible motion with long-term consistency. When generating extended sequences, the model's internal representation of the subject's identity weakens over time. As Roberto Henschel et al. observe in StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text, without an explicit long-term reference anchor, cumulative errors across successive segments cause the model to lose track of the original subject's specific visual characteristics.

Furthermore, there is an inherent competition within the network between motion priors (which dictate how things move) and identity constraints (which dictate what things look like). When a model is driven by strong external signals—such as complex text prompts, trajectories, or audio tracks—the motion layers can override the appearance layers, causing textures to warp and features to dissolve.

To combat this, researchers have developed three core strategies: decoupling space from time, forcing reference reliance through information bottlenecks, and tracking explicit spatiotemporal features.

Decoupling Spatial Identity from Temporal Motion

If generating motion and identity simultaneously causes them to conflict, an intuitive solution is to separate them entirely. By anchoring the spatial identity first, the model can treat the subsequent generation process purely as a problem of animating a fixed asset.

For example, Yuji Wang et al. demonstrate in Identity-Preserving Text-to-Video Generation Guided by Simple yet Effective Spatial-Temporal Decoupled Representations that generating a high-fidelity static image before applying temporal transformations prevents the character's features from dissolving. By establishing a dedicated mechanism to anchor spatial identity first, the downstream temporal layers are constrained to manipulate geometry rather than reinventing it.

Similarly, in the context of video editing and inpainting, Ming Xie et al. introduce a framework in VidSplice: Towards Coherent Video Inpainting via Explicit Spaced Frame Guidance that decouples the task into multi-frame consistent image generation followed by masked area motion propagation. By using sparsely sampled "anchor frames" as global spatiotemporal cues, the system maintains high-fidelity object identity and background stability across long durations. Even when dealing with highly abstract inputs, Jiayi Zheng isolates character identity from background synthesis in FairyGen: Storied Cartoon Video from a Single Child-Drawn Character, using 3D proxies to ensure that an abstract character remains recognizable across distinct narrative shots.

Forcing Reference Reliance via Information Bottlenecks

Neural networks are inherently lazy: when predicting the next frame, the easiest way to minimize short-term loss is to copy and slightly warp the immediate history frames. However, this short-term shortcut is exactly what causes long-term identity drift.

To break this habit, models must be forced to look back at the original reference image for identity, while using the recent past only for motion continuity.

[Reference Image] ──────────────────────────► [Generative Model] ◄─── [Target Frame]
                                                     ▲
[History Frames] ──► [Noise & Patch Drop] ───────────┘
                      (Destroys Identity,
                       Preserves Motion)

The authors of Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animation, led by Jiahao Cui, implement this intuition via a clever information bottleneck. During training, they intentionally corrupt the input history frames using random patch-dropping and Gaussian noise. This corruption destroys fine-grained identity details in the history but preserves the macro-level motion cues. Consequently, the model cannot rely on the immediate past to figure out who it is drawing; it is forced to extract identity information exclusively from the static reference image, effectively halting identity drift over long durations.

Another way to anchor the model to the source is through residual learning. Instead of predicting each frame from scratch, Zhongwei Zhang et al. propose a framework in TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models that uses an image noise prior to anchor animated frames directly to the original static image. By predicting noise in a residual fashion, the model explicitly separates the static reference state from the motion-induced changes, preventing textures from warping or sliding away from the source identity.

Dense Tracking and State Estimation

A final frontier in preventing identity drift is providing the model with explicit geometric awareness of the objects it is animating. Without structural priors, a model does not understand that a texture belongs to a specific surface, leading to "floating" features or sudden morphing.

To ground the model geometrically, Hyeonho Jeong et al. introduce point-tracking supervision during the training phase in Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation. By forcing the diffusion model to learn how explicit points move across frames, the network naturally learns to preserve shapes and textures over time, significantly reducing appearance drift.

Without Tracking:  [Frame t: Texture A] ──(Drifts)──► [Frame t+1: Texture A']
With Tracking:     [Frame t: Point P1]  ──(Tracked)─► [Frame t+1: Point P1]
                   (Texture pinned to P1)             (Texture remains identical)

For highly sensitive tasks like video face restoration, models can even borrow principles from classical control theory. Chongyi Li et al. develop a framework called KEEP in Kalman-Inspired Feature Propagation for Video Face Super-Resolution, which uses Kalman filtering principles to maintain a stable face prior over time. By recurrently updating latent features using a learned "Kalman Gain," the model continuously balances the current frame's observed data with the accumulated historical identity state. This mathematical regulation ensures that restored facial features do not drift in shape or appearance, maintaining strict continuity even through rapid motion or occlusions.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 300 papers
  2. Version 1 · 2026-10-04 · 246 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_identity_drift_5211c97f,
  title = {identity drift},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/identity-drift/rdr_5211c97fa7674e05bcdfb4ce5d51f7a1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_identity_drift_5211c97f_v2,
  title = {identity drift},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/identity-drift/rdr_5211c97fa7674e05bcdfb4ce5d51f7a1/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake