ScoutCache: Efficient Few-Step Autoregressive Video Diffusion via Conditional Readout and Selective Caching

Hao Tan, Jinghui Qin, Xuming HuarXiv 2026 · arXiv

TL;DR

ScoutCache is a training-free framework that significantly accelerates autoregressive video diffusion by optimizing denoising, historical context reading, and cache construction. By skipping redundant deep-network computations and selectively caching only the most informative spatial tokens, it achieves up to a 2.28× speedup while actually improving video generation quality.

What Problem Does This Paper Address?

Generating long, high-quality videos typically requires autoregressive diffusion models that synthesize video "chunks" one after another. Even with recent "few-step" diffusion techniques, inference remains slow due to three main bottlenecks:

  1. Repeated Denoising: Each chunk requires multiple passes through a deep Transformer network.
  2. Historical KV Reading: To maintain temporal consistency, the model must attend to a massive "Key-Value" (KV) cache of previously generated frames, which is computationally expensive.
  3. Cache Construction: After generating a chunk, the model performs an extra forward pass just to build the KV cache for the next chunk.

Existing acceleration methods often focus on only one of these areas or require expensive retraining. ScoutCache aims to solve all three simultaneously without modifying the underlying model's parameters.

Methods

ScoutCache introduces three core components that work together during the inference process:

  • Conditional Denoising Readout (CDR): The authors observed that changes in the network's shallow features correlate strongly with the final denoising update. CDR exploits this by running only the first Transformer layer during intermediate denoising steps and using a calibrated "readout" mapping to estimate the full update, bypassing the deeper layers.
  • Output Risk Guided Historical KV Reading (ORHR): Not every attention head needs to see every past frame. ORHR measures the "risk" (output perturbation) of removing specific historical frames. It then allocates a fixed "reading budget," ensuring each head only accesses the most critical past information.
  • Content and Coverage Guided Sparse Cache Construction (CCSC): Instead of building a dense KV cache for every pixel, CCSC selects spatial tokens that have changed significantly or haven't been updated recently. It uses a "multiplicity weighted attention" mechanism to ensure these sparse tokens correctly represent their neighboring areas during future attention steps.
Figure 2. The top shows the overall inference pipeline, integrating denoising acceleration and KV cache optimization. The bottom illustrates three components. (a) CDR estimates clean latent updates from noise variations and shallow feature changes. (b) ORHR allocates historical KV reading budgets based on attention output perturbations. (c) CCSC reduces cache construction costs through content and coverage guided spatial token selection.

Datasets

The authors evaluated ScoutCache using two primary backbones, Self-Forcing and LongLive (both 1.3B parameter models), across two major benchmarks:

  • VBench: Used for 5-second video generation across 946 prompts to measure quality across 16 dimensions.
  • MovieGen: Used to test long-form generation (30s, 60s, and 120s) using 128 extended prompts.

Key Insights and Ideas

  • Shallow Features are Predictive: The most surprising insight is that the first layer of a Transformer captures enough "trajectory" information to predict what the subsequent 30+ layers would have done during a denoising step.
  • Attention Stability: While different layers and heads have wildly different dependencies on historical context, these patterns remain remarkably stable across video chunks. This allows ScoutCache to calibrate its strategy on the first few chunks and reuse it for the rest of the video.
  • Less is More for Quality: Interestingly, selective historical reading (ORHR) can actually improve quality. By reducing "previous-frame bias"—where the model gets stuck repeating old poses—ScoutCache allows for more fluid and natural motion progression.
Figure 3. Qualitative comparison of video generation results. We show four sampled frames for each prompt, comparing ScoutCache with the original backbones, Dummy Forcing, and Forcing-KV.

Key Results

  • Speedups: ScoutCache achieved a 2.28× speedup on 120-second MovieGen videos using the Self-Forcing backbone and a 1.86× speedup with LongLive.
  • Efficiency Gains: The framework reduced denoising time by ~25% and KV cache construction time by ~19% compared to standard "Dummy Forcing" baselines.
  • Quality Improvements: On MovieGen (120s), the Total Score improved from 75.03 to 79.11. Qualitative results show that ScoutCache maintains clearer contours and better scene structure than competing acceleration methods like Forcing-KV.
  • Training-Free: All these gains are achieved without any fine-tuning or architectural changes, making it a "plug-and-play" solution for existing models.

Limitations and Future Work

  • Calibration Overhead: ScoutCache requires the first three chunks of a video to be processed normally to calibrate the CDR and ORHR modules. While this overhead is amortized over long videos, it is less efficient for very short clips.
  • Fixed Budget Heuristics: The current implementation uses a fixed global budget for historical reading. The authors suggest that dynamically adjusting this budget based on the complexity of the video content could yield further improvements.
  • Hardware Specifics: While tested on A100 GPUs, the actual wall-clock speedup may vary depending on the specific memory bandwidth and compute capabilities of different hardware architectures.

Summary updated

Abstract

Few-step autoregressive video diffusion enables efficient long video generation, but inference still incurs substantial computational costs from repeated denoising and KV cache operations. Existing acceleration methods reduce parts of this overhead, while historical KV reading and cache construction remain costly across video chunks. To address these challenges, we propose ScoutCache, a training-free framework that jointly optimizes denoising, historical KV reading, and cache construction. We observe that shallow feature changes provide useful information for estimating clean latent updates, while historical KV dependence varies across Transformer layers and attention heads but remains relatively stable across video chunks. Based on these observations, we develop Conditional Denoising Readout (CDR) to estimate intermediate denoising updates from shallow feature changes and known noise variations. We further introduce Output Risk Guided Historical KV Reading (ORHR) to allocate historical reading budgets based on attention output perturbations, and Content and Coverage Guided Sparse Cache Construction (CCSC) to reduce cache construction costs by selecting informative spatial tokens while maintaining spatial coverage. Extensive experiments on Self-Forcing and LongLive using VBench and MovieGen demonstrate that ScoutCache achieves up to 2.28$\times$ and 1.86$\times$ speedups, respectively, while consistently improving generation quality without modifying pretrained model parameters.
BibTeX
@misc{tan2026scoutcache,
  author = {Hao Tan and Jinghui Qin and Xuming Hu},
  title = {{ScoutCache: Efficient Few-Step Autoregressive Video Diffusion via Conditional Readout and Selective Caching}},
  year = {2026},
  eprint = {2611.21038},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21038},
}
Figure