TL;DR
LongLive-RAG introduces a general retrieval-augmented generation framework for autoregressive video diffusion, treating previously generated latents as a dynamic, searchable history. By actively retrieving relevant past context rather than relying solely on a local sliding window, it prevents identity drift and error accumulation over long horizons. This lightweight approach achieves state-of-the-art consistency in long video generation with negligible computational overhead.
What Problem Does This Paper Address?
Autoregressive (AR) video diffusion models generate long videos block-by-block, making them highly flexible for variable-length synthesis. However, to keep computational costs bounded, these models typically employ a sliding-window attention mechanism, meaning they only look at the most recently generated frames.
This creates an irreversible generation trajectory: if the model introduces a slight artifact, color shift, or appearance error within the active window, all subsequent frames are forced to condition on this degraded context. Over long durations (tens or hundreds of seconds), this leads to severe error accumulation, background morphing, and "identity drift"—where characters or scenes completely change their appearance over time.
Methods
Instead of relying on a rigid sliding window or heavily compressed summaries of the past, LongLive-RAG treats the self-generated video history as a content-addressable memory bank.
The framework operates via three main components:
- Latent Space Retrieval: Rather than decoding latents to raw pixels to perform a search (which is computationally expensive and breaks the diffusion pipeline), a lightweight encoder maps the generated latents into a compact 1024-dimensional embedding space.
- Window Temporal Delta Loss: Standard autoencoders optimized purely for reconstruction make temporally adjacent frames look nearly identical. This would cause a top-$K$ retrieval mechanism to waste its budget fetching redundant frames already present in the local sliding window. To combat this, the authors introduce a novel loss that penalizes excessive similarity within a short temporal window, forcing the embedding space to focus on distinct, long-range changes.
- Smoothness Penalty: A second-order smoothness loss is added to ensure that the embedding trajectory remains stable and organized over time, preventing noisy or erratic context switching.
During inference, the system uses the latest completed latent block as a query to perform a cosine-similarity search over the historical pool (excluding the immediate past via a recency guard). The top-$K$ retrieved historical latents are then appended directly into the generator's attention context alongside the local sliding window frames.
Datasets
The authors evaluated LongLive-RAG using all 128 prompts from MovieGenBench, with prompts refined via Qwen2.5-7B-Instruct to ensure high-quality generation baselines.
To train the retrieval encoder offline, they constructed a dataset of over 20,000 training latents by generating 90-latent rollouts from a random 10% sample of the Self-Forcing prompt pool using a frozen LongLive generator.
Key Insights and Ideas
- Memory over Compression: Existing long-video methods often compress past history into fixed summary tokens, which permanently discards fine-grained visual details. LongLive-RAG uses compact representations only as search keys, but retrieves the original, uncompressed latents for the generator to attend to, preserving native visual details.
- The Anti-Collapse Necessity: Relying on reconstruction loss alone creates a massive high-similarity band around the diagonal of the video timeline. The Window Temporal Delta Loss acts as an essential anti-collapse mechanism that clears out local redundancy, enabling true non-local context retrieval.
- Latent vs. Pixel Search: Searching directly in the VAE latent space avoids the heavy computational overhead of VAE decoding/encoding cycles at every step and keeps the features perfectly aligned with what the denoising transformer expects.
Key Results
- State-of-the-Art Consistency: LongLive-RAG achieved the best average VBench-Long rank across three distinct autoregressive backbones (Causal-Forcing, Self-Forcing, and LongLive) and three generation lengths (30s, 60s, and 120s). It significantly boosted subject consistency, background consistency, and imaging quality.
- Negligible Overhead: The retrieval mechanism adds only 4.08 ms per block (totaling just 490 ms of overhead for an entire 120-second video rollout). The process is heavily dominated by latent encoding (3.96 ms), while the top-$K$ similarity search takes a mere 0.08 ms.
Limitations and Future Work
LongLive-RAG is designed to operate on top of a frozen base generator checkpoint. While it drastically improves how a model selects and reuses its self-generated history, it does not alter the underlying generation capabilities of the base video transformer itself. Consequently, the absolute visual quality, motion dynamics, and synthesis limits remain bounded by the performance ceiling of the base AR model.
Summary updated