TL;DR
SEMANTICKV is a training-free KV cache compression method that uses a planner's semantic constraints to decide which parts of a video's history are worth keeping in memory. By prioritizing states that support specific requested actions or attributes, it achieves a 3.35× reduction in KV cache memory and a 1.22× speedup while maintaining or even improving semantic fidelity compared to a full, uncompressed cache.
What Problem Does This Paper Address?
Autoregressive (AR) video generation has moved toward "agentic" workflows: a semantic planner refines a prompt into specific constraints, a generator produces candidates, and a critic verifies them. While powerful, this process is computationally expensive. As videos get longer, the Key-Value (KV) cache—which stores historical states to avoid recomputation—grows massive, leading to high VRAM usage and slower inference.
Existing KV cache compression methods (like R-KV or Dummy Forcing) are "semantic-blind." They use global importance scores to prune the cache. The danger here is "semantic dilution": a frame that is vital for a specific, brief action (e.g., "dunking a basketball") might be assigned a low average importance score because it isn't relevant to the rest of the video. When these frames are discarded, the model loses the "memory" needed to satisfy the user's specific requests, leading to hallucinations or loss of detail.
Methods
The authors introduce SEMANTICKV, the first compression method designed to align cache retention with the explicit goals of an agentic planner.
The core algorithm follows a three-step process:
- Constraint-to-Position Mapping: It matches semantic constraints (e.g., "Sailboat," "Moves right") to specific words in the prompt. It then uses the generator's own text cross-attention to identify which video positions are currently attending to those words.
- Semantic Importance Scoring: Self-attention queries from these identified positions are collected into constraint-specific groups. For every entry in the KV cache, the method calculates a score by averaging attention weights within each group (to prevent bias toward groups with more queries) and then taking the maximum across all groups. This ensures that if a cached entry is strongly used by any single constraint, it receives a high priority.
- Budgeted Retention: A single top-$k$ selection retains the highest-scoring entries within a memory budget. Crucially, this is a training-free, "plug-and-play" approach that doesn't require retraining the video backbone.
Datasets
The researchers evaluated their method across four AR video backbones (LongLive, LongLive 2.0, Self Forcing, and Causal Forcing) using three major benchmarks:
- VBench: 946 prompts for 5-second video quality and semantic fidelity.
- VBenchLong: 128 prompts from MovieGen for 30-second and 60-second long-horizon consistency.
- VisionReward: A 100-prompt set for human-preference-based assessment.
Key Insights and Ideas
- The "Max" Aggregation: The decision to take the maximum score across semantic groups is mathematically grounded. The authors prove that this selection minimizes an additive upper bound on the "discarded attention" for the worst-affected semantic group.
- Semantics as a Guide for Memory: The paper argues that the same semantics used to guide generation should also guide what the model remembers. By explicitly linking the KV cache to the planner's constraints, they bridge the gap between high-level planning and low-level transformer mechanics.
- Quality Gains through Compression: Surprisingly, SEMANTICKV sometimes outperforms the "Full KV" (uncompressed) baseline in semantic fidelity. This suggests that by pruning "distractor" states and focusing on constraint-relevant history, the model can actually follow instructions more accurately.
Key Results
The quantitative results demonstrate a significant efficiency-quality trade-off:
- Memory Efficiency: On the Causal Forcing backbone, it reduced KV cache memory from 6.04 GB to 1.81 GB (a 3.35× reduction).
- Throughput: It achieved a 1.22× speedup in frames per second (FPS) over the uncompressed baseline.
- Semantic Fidelity: On VBench, SEMANTICKV achieved a Total Score of 84.10, surpassing both the strongest compressed baseline (84.05) and the Full KV reference (83.99).
- Long-Horizon Performance: For 60-second videos, it maintained high quality while providing a 1.29× speedup, proving the method scales to extended generation.
Qualitatively, the method excels at preserving specific objects and actions that other methods fail to capture, such as a person successfully dunking a basketball rather than just dribbling.
Limitations and Future Work
- Layer-wise Allocation: Currently, the method allocates the memory budget across transformer layers proportionally to their original size. The authors suggest that non-uniform allocation (giving more budget to layers that are more "semantic") could yield further gains.
- Planner Dependency: The performance is inherently tied to the quality of the semantic planner. If the planner fails to extract a key constraint, SEMANTICKV won't know to protect the associated cache entries.
- Constraint Satisfaction: While the method preserves the history needed to satisfy constraints, it doesn't guarantee that the generator will actually satisfy them in the future frames—it only ensures the "tools" to do so remain in the cache.
Summary updated