LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generation

Qixin Hu, Shuai Yang, Wei Huang, Song Han, Yukang ChenarXiv 2026 · arXiv

TL;DR

LongLive-RAG introduces a general retrieval-augmented generation framework for autoregressive video diffusion, treating previously generated latents as a dynamic, searchable history. By actively retrieving relevant past context rather than relying solely on a local sliding window, it prevents identity drift and error accumulation over long horizons. This lightweight approach achieves state-of-the-art consistency in long video generation with negligible computational overhead.

What Problem Does This Paper Address?

Autoregressive (AR) video diffusion models generate long videos block-by-block, making them highly flexible for variable-length synthesis. However, to keep computational costs bounded, these models typically employ a sliding-window attention mechanism, meaning they only look at the most recently generated frames.

This creates an irreversible generation trajectory: if the model introduces a slight artifact, color shift, or appearance error within the active window, all subsequent frames are forced to condition on this degraded context. Over long durations (tens or hundreds of seconds), this leads to severe error accumulation, background morphing, and "identity drift"—where characters or scenes completely change their appearance over time.

Methods

Instead of relying on a rigid sliding window or heavily compressed summaries of the past, LongLive-RAG treats the self-generated video history as a content-addressable memory bank.

Overview of the LongLive-RAG framework, detailing the generation pipeline, latent similarity selection, and error trajectory correction.

The framework operates via three main components:

  1. Latent Space Retrieval: Rather than decoding latents to raw pixels to perform a search (which is computationally expensive and breaks the diffusion pipeline), a lightweight encoder maps the generated latents into a compact 1024-dimensional embedding space.
  2. Window Temporal Delta Loss: Standard autoencoders optimized purely for reconstruction make temporally adjacent frames look nearly identical. This would cause a top-$K$ retrieval mechanism to waste its budget fetching redundant frames already present in the local sliding window. To combat this, the authors introduce a novel loss that penalizes excessive similarity within a short temporal window, forcing the embedding space to focus on distinct, long-range changes.
  3. Smoothness Penalty: A second-order smoothness loss is added to ensure that the embedding trajectory remains stable and organized over time, preventing noisy or erratic context switching.

During inference, the system uses the latest completed latent block as a query to perform a cosine-similarity search over the historical pool (excluding the immediate past via a recency guard). The top-$K$ retrieved historical latents are then appended directly into the generator's attention context alongside the local sliding window frames.

Datasets

The authors evaluated LongLive-RAG using all 128 prompts from MovieGenBench, with prompts refined via Qwen2.5-7B-Instruct to ensure high-quality generation baselines.

To train the retrieval encoder offline, they constructed a dataset of over 20,000 training latents by generating 90-latent rollouts from a random 10% sample of the Self-Forcing prompt pool using a frozen LongLive generator.

Key Insights and Ideas

  • Memory over Compression: Existing long-video methods often compress past history into fixed summary tokens, which permanently discards fine-grained visual details. LongLive-RAG uses compact representations only as search keys, but retrieves the original, uncompressed latents for the generator to attend to, preserving native visual details.
  • The Anti-Collapse Necessity: Relying on reconstruction loss alone creates a massive high-similarity band around the diagonal of the video timeline. The Window Temporal Delta Loss acts as an essential anti-collapse mechanism that clears out local redundancy, enabling true non-local context retrieval.
Embedding-space analysis showing how the temporal delta loss and smoothing stabilize the trajectory and reduce local redundancy.
  • Latent vs. Pixel Search: Searching directly in the VAE latent space avoids the heavy computational overhead of VAE decoding/encoding cycles at every step and keeps the features perfectly aligned with what the denoising transformer expects.

Key Results

  • State-of-the-Art Consistency: LongLive-RAG achieved the best average VBench-Long rank across three distinct autoregressive backbones (Causal-Forcing, Self-Forcing, and LongLive) and three generation lengths (30s, 60s, and 120s). It significantly boosted subject consistency, background consistency, and imaging quality.
  • Negligible Overhead: The retrieval mechanism adds only 4.08 ms per block (totaling just 490 ms of overhead for an entire 120-second video rollout). The process is heavily dominated by latent encoding (3.96 ms), while the top-$K$ similarity search takes a mere 0.08 ms.

Limitations and Future Work

LongLive-RAG is designed to operate on top of a frozen base generator checkpoint. While it drastically improves how a model selects and reuses its self-generated history, it does not alter the underlying generation capabilities of the base video transformer itself. Consequently, the absolute visual quality, motion dynamics, and synthesis limits remain bounded by the performance ceiling of the base AR model.

Summary updated

Abstract

Autoregressive (AR) video diffusion enables variable-length synthesis, but long-horizon generation often suffers from accumulated errors and identity drift. For efficiency, existing methods commonly adopt sliding-window attention during generation. This creates an irreversible generation trajectory: once the active window accumulates appearance errors, subsequent generations can only condition on this degraded trajectory and drift further away. We address this limitation by formulating long video generation as a retrieval-augmented generation (RAG) problem. Rather than relying solely on the recent window, we treat previously generated latents as a dynamic, searchable history. We propose LongLive-RAG, a general retrieval framework for AR video generation. At each new block, LongLive-RAG uses a query embedding to retrieve relevant historical latents. This lightweight retrieval step adds only a small overhead relative to generation and lets the generator condition on non-local context instead of only the recent window. To make retrieval more discriminative, we introduce the Window Temporal Delta Loss that suppresses redundant local similarity and encourages embeddings to capture meaningful temporal changes. Together, these components help reduce error accumulation caused by sliding-window attention. Experiments across multiple AR backbones and generation lengths show improved long-video quality and the best average VBench-Long rank. To our knowledge, among open-ended AR long video generation methods, LongLive-RAG is the first to formulate self-generated latent history as content-addressable retrieval memory. Code is available at https://github.com/qixinhu11/LongLive-RAG.
BibTeX
@misc{hu2026longlive,
  author = {Qixin Hu and Shuai Yang and Wei Huang and Song Han and Yukang Chen},
  title = {{LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generation}},
  year = {2026},
  eprint = {2606.02553},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.02553},
}

Loading related content...

Figure