TL;DR
ScoutCache is a training-free framework that significantly accelerates autoregressive video diffusion by optimizing denoising, historical context reading, and cache construction. By skipping redundant deep-network computations and selectively caching only the most informative spatial tokens, it achieves up to a 2.28× speedup while actually improving video generation quality.
What Problem Does This Paper Address?
Generating long, high-quality videos typically requires autoregressive diffusion models that synthesize video "chunks" one after another. Even with recent "few-step" diffusion techniques, inference remains slow due to three main bottlenecks:
- Repeated Denoising: Each chunk requires multiple passes through a deep Transformer network.
- Historical KV Reading: To maintain temporal consistency, the model must attend to a massive "Key-Value" (KV) cache of previously generated frames, which is computationally expensive.
- Cache Construction: After generating a chunk, the model performs an extra forward pass just to build the KV cache for the next chunk.
Existing acceleration methods often focus on only one of these areas or require expensive retraining. ScoutCache aims to solve all three simultaneously without modifying the underlying model's parameters.
Methods
ScoutCache introduces three core components that work together during the inference process:
- Conditional Denoising Readout (CDR): The authors observed that changes in the network's shallow features correlate strongly with the final denoising update. CDR exploits this by running only the first Transformer layer during intermediate denoising steps and using a calibrated "readout" mapping to estimate the full update, bypassing the deeper layers.
- Output Risk Guided Historical KV Reading (ORHR): Not every attention head needs to see every past frame. ORHR measures the "risk" (output perturbation) of removing specific historical frames. It then allocates a fixed "reading budget," ensuring each head only accesses the most critical past information.
- Content and Coverage Guided Sparse Cache Construction (CCSC): Instead of building a dense KV cache for every pixel, CCSC selects spatial tokens that have changed significantly or haven't been updated recently. It uses a "multiplicity weighted attention" mechanism to ensure these sparse tokens correctly represent their neighboring areas during future attention steps.
Datasets
The authors evaluated ScoutCache using two primary backbones, Self-Forcing and LongLive (both 1.3B parameter models), across two major benchmarks:
- VBench: Used for 5-second video generation across 946 prompts to measure quality across 16 dimensions.
- MovieGen: Used to test long-form generation (30s, 60s, and 120s) using 128 extended prompts.
Key Insights and Ideas
- Shallow Features are Predictive: The most surprising insight is that the first layer of a Transformer captures enough "trajectory" information to predict what the subsequent 30+ layers would have done during a denoising step.
- Attention Stability: While different layers and heads have wildly different dependencies on historical context, these patterns remain remarkably stable across video chunks. This allows ScoutCache to calibrate its strategy on the first few chunks and reuse it for the rest of the video.
- Less is More for Quality: Interestingly, selective historical reading (ORHR) can actually improve quality. By reducing "previous-frame bias"—where the model gets stuck repeating old poses—ScoutCache allows for more fluid and natural motion progression.
Key Results
- Speedups: ScoutCache achieved a 2.28× speedup on 120-second MovieGen videos using the Self-Forcing backbone and a 1.86× speedup with LongLive.
- Efficiency Gains: The framework reduced denoising time by ~25% and KV cache construction time by ~19% compared to standard "Dummy Forcing" baselines.
- Quality Improvements: On MovieGen (120s), the Total Score improved from 75.03 to 79.11. Qualitative results show that ScoutCache maintains clearer contours and better scene structure than competing acceleration methods like Forcing-KV.
- Training-Free: All these gains are achieved without any fine-tuning or architectural changes, making it a "plug-and-play" solution for existing models.
Limitations and Future Work
- Calibration Overhead: ScoutCache requires the first three chunks of a video to be processed normally to calibrate the CDR and ORHR modules. While this overhead is amortized over long videos, it is less efficient for very short clips.
- Fixed Budget Heuristics: The current implementation uses a fixed global budget for historical reading. The authors suggest that dynamically adjusting this budget based on the complexity of the video content could yield further improvements.
- Hardware Specifics: While tested on A100 GPUs, the actual wall-clock speedup may vary depending on the specific memory bandwidth and compute capabilities of different hardware architectures.
Summary updated