Anchor Frame Attention for Consistent Video Generation

94 concepts64 papers1 proposal

When adapting powerful image diffusion models for video generation or editing, the primary obstacle is not creating realistic imagery, but maintaining visual consistency across time. Because image models denoise latents independently or within narrow temporal windows, they suffer from two related phenomena: temporal flickering (rapid, jarring changes in texture and lighting) and identity drift (the gradual morphing of a character's features or environment over long sequences).

To understand why this happens, consider a standard video generation pipeline where each frame primarily attends to itself or its immediate predecessor. While this local connection ensures smooth frame-to-frame transitions, it lacks a global compass. Small errors in texture or geometry accumulate over time, leading to an inescapable "death spiral" of quality degradation.

The breakthrough concept across recent video synthesis frameworks is anchor-frame attention (also known as cross-frame attention injection or temporal identity anchoring). Instead of allowing the model's attention mechanism to drift freely, these methods designate one or more high-quality frames as permanent visual "anchors" to lock down the identity and style of the entire sequence.

The Core Mechanism: Pinning the Latent Space

To build intuition for why anchor-frame attention works, let us look at how attention operates under the hood. In a standard self-attention layer, a frame generates Queries ($Q$), Keys ($K$), and Values ($V$) from its own features:

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V$$

The Keys and Values encode the appearance and structural details of the scene, while the Queries determine where those details should be placed.

In anchor-frame attention, a subsequent frame $t$ still generates its own Queries ($Q_t$), but it computes its attention matrix using the Keys ($K_{\text{anchor}}$) and Values ($V_{\text{anchor}}$) extracted from the anchor frame (often the very first frame, $t=0$).

By forcing the model to retrieve appearance information from a fixed source, the latent space is effectively locked. The Queries of the current frame dictate the new motion or pose, but the actual textures, colors, and identity markers are pulled directly from the anchor. This simple mathematical constraint eliminates flickering because the underlying visual assets remain identical across the entire sequence.

Architectural Variations: Sinks, Chains, and Skeletons

Researchers have deployed this anchoring principle in several distinct configurations depending on the specific requirements of the task:

  • Permanent Attention Sinks (All-to-First): For long-form or autoregressive video generation, treating the initial frame as a permanent "attention sink" is highly effective. As shown by Wang et al. in EchoShot and Shin et al. in MotionStream, keeping the first frame permanently inside the attention window prevents cumulative drift over hundreds of frames. Similarly, Yang et al. in LongLive and Wang et al. in REST leverage this persistent anchoring to achieve real-time streaming generation while keeping character identities rock-solid.
  • Dual-Reference Attention: Relying solely on the first frame can sometimes restrict fluid motion if the subject deforms significantly. To balance global stability with local continuity, frameworks like MotionCraft (Montanaro et al.) and Rerender A Video (Yang et al.) implement a hybrid approach. Each frame queries both the initial frame (for structural anchors) and the immediately preceding frame (for local smoothness). Conversely, Khandelwal in InFusion demonstrates that sequential anchoring to the immediate predecessor alone can drastically reduce flickering in zero-shot settings by preventing divergent creative choices between adjacent frames.
  • Spatiotemporal Skeletons: For long-duration videos or complex multi-event narratives where a single first frame is insufficient, researchers distribute multiple anchors across time. In VidSplice, Xie et al. utilize sparsely sampled anchor frames to act as a spatiotemporal skeleton, ensuring background stability and identity preservation across extended sequences. Oh et al. apply a similar logic in MEVG by using the final frame of a preceding video segment as the structural and textural anchor for the next event block, preventing jarring shifts during narrative transitions.

Training-Free Versatility and Applications

A major advantage of anchor-frame attention is that it can often be applied in a training-free (zero-shot) manner. Because the mechanism modifies how existing attention maps interact rather than introducing new weights, it can be retrofitted directly into pretrained text-to-image or text-to-video models.

For example, in zero-shot video editing, Ceylan et al. in Pix2Video inject self-attention features from an edited anchor frame to propagate stylistic changes consistently without per-video finetuning. In the domain of 3D and 4D generation, Zeng et al. introduce STAG4D, which fuses anchor-frame keys and values into subsequent views to prevent texture flickering during the diffusion process without requiring additional training cycles.

This paradigm also extends to tasks requiring precise structural guidance. In Audio-Synchronized Visual Animation, Zhang et al. utilize first-frame lookups within spatial and temporal attention layers to animate a static image using audio cues, ensuring the subject's identity never morphs or degrades as it moves to the rhythm. By re-routing the flow of information back to a trusted visual source, anchor-frame attention transforms isolated image generators into cohesive video synthesizers.

Key Foundational Work

Video Object Segmentation Using Global and Instance Embedding Learning (2021) — contributed 1 concept elements to this direction.
StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation (2022) — contributed 1 concept elements to this direction.
Word-Level Fine-Grained Story Visualization (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_anchor_frame_attention_411cd250,
  title = {anchor-frame attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/anchor-frame-attention/rdr_411cd250f5734f3fb7d4531fd1a1ba6e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_anchor_frame_attention_411cd250_v1,
  title = {anchor-frame attention},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/anchor-frame-attention/rdr_411cd250f5734f3fb7d4531fd1a1ba6e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake