Semantically Aligned KV Cache Compression for Efficient Agentic Autoregressive Video Generation

Weizhong Huang, Huaijia Kou, Tianyu Xie, Xiawu Zheng, Liujuan CaoarXiv 2026 · arXiv

TL;DR

SEMANTICKV is a training-free KV cache compression method that uses a planner's semantic constraints to decide which parts of a video's history are worth keeping in memory. By prioritizing states that support specific requested actions or attributes, it achieves a 3.35× reduction in KV cache memory and a 1.22× speedup while maintaining or even improving semantic fidelity compared to a full, uncompressed cache.

What Problem Does This Paper Address?

Autoregressive (AR) video generation has moved toward "agentic" workflows: a semantic planner refines a prompt into specific constraints, a generator produces candidates, and a critic verifies them. While powerful, this process is computationally expensive. As videos get longer, the Key-Value (KV) cache—which stores historical states to avoid recomputation—grows massive, leading to high VRAM usage and slower inference.

Existing KV cache compression methods (like R-KV or Dummy Forcing) are "semantic-blind." They use global importance scores to prune the cache. The danger here is "semantic dilution": a frame that is vital for a specific, brief action (e.g., "dunking a basketball") might be assigned a low average importance score because it isn't relevant to the rest of the video. When these frames are discarded, the model loses the "memory" needed to satisfy the user's specific requests, leading to hallucinations or loss of detail.

Methods

The authors introduce SEMANTICKV, the first compression method designed to align cache retention with the explicit goals of an agentic planner.

Agentic AR video generation workflow.

The core algorithm follows a three-step process:

  1. Constraint-to-Position Mapping: It matches semantic constraints (e.g., "Sailboat," "Moves right") to specific words in the prompt. It then uses the generator's own text cross-attention to identify which video positions are currently attending to those words.
  2. Semantic Importance Scoring: Self-attention queries from these identified positions are collected into constraint-specific groups. For every entry in the KV cache, the method calculates a score by averaging attention weights within each group (to prevent bias toward groups with more queries) and then taking the maximum across all groups. This ensures that if a cached entry is strongly used by any single constraint, it receives a high priority.
  3. Budgeted Retention: A single top-$k$ selection retains the highest-scoring entries within a memory budget. Crucially, this is a training-free, "plug-and-play" approach that doesn't require retraining the video backbone.
Figure 2: Overview of SEMANTICKV. Semantic constraints R are matched to prompt tokens. Text cross-attention identifies related video positions, from which we collect the generator’s self-attention queries. Their attention to each KV entry is averaged within groups and maximized across groups. A single top-k selection retains high-scoring entries within the byte budget, together with states required by the backbone. Retained keys, values, and positions remain unchanged.

Datasets

The researchers evaluated their method across four AR video backbones (LongLive, LongLive 2.0, Self Forcing, and Causal Forcing) using three major benchmarks:

  • VBench: 946 prompts for 5-second video quality and semantic fidelity.
  • VBenchLong: 128 prompts from MovieGen for 30-second and 60-second long-horizon consistency.
  • VisionReward: A 100-prompt set for human-preference-based assessment.

Key Insights and Ideas

  • The "Max" Aggregation: The decision to take the maximum score across semantic groups is mathematically grounded. The authors prove that this selection minimizes an additive upper bound on the "discarded attention" for the worst-affected semantic group.
  • Semantics as a Guide for Memory: The paper argues that the same semantics used to guide generation should also guide what the model remembers. By explicitly linking the KV cache to the planner's constraints, they bridge the gap between high-level planning and low-level transformer mechanics.
  • Quality Gains through Compression: Surprisingly, SEMANTICKV sometimes outperforms the "Full KV" (uncompressed) baseline in semantic fidelity. This suggests that by pruning "distractor" states and focusing on constraint-relevant history, the model can actually follow instructions more accurately.

Key Results

The quantitative results demonstrate a significant efficiency-quality trade-off:

  • Memory Efficiency: On the Causal Forcing backbone, it reduced KV cache memory from 6.04 GB to 1.81 GB (a 3.35× reduction).
  • Throughput: It achieved a 1.22× speedup in frames per second (FPS) over the uncompressed baseline.
  • Semantic Fidelity: On VBench, SEMANTICKV achieved a Total Score of 84.10, surpassing both the strongest compressed baseline (84.05) and the Full KV reference (83.99).
  • Long-Horizon Performance: For 60-second videos, it maintained high quality while providing a 1.29× speedup, proving the method scales to extended generation.

Qualitatively, the method excels at preserving specific objects and actions that other methods fail to capture, such as a person successfully dunking a basketball rather than just dribbling.

Table 1: Five-second VBench at compression ratio 0.7: Total Score, temporal quality, and six shared semantic diagnostics. Table 3 gives all 16 dimensions. Full KV is the uncompressed reference; efficiency is DiT-only. ↑/↓: higher/lower is better. Bold: best compressed result for Total Score.

Limitations and Future Work

  • Layer-wise Allocation: Currently, the method allocates the memory budget across transformer layers proportionally to their original size. The authors suggest that non-uniform allocation (giving more budget to layers that are more "semantic") could yield further gains.
  • Planner Dependency: The performance is inherently tied to the quality of the semantic planner. If the planner fails to extract a key constraint, SEMANTICKV won't know to protect the associated cache entries.
  • Constraint Satisfaction: While the method preserves the history needed to satisfy constraints, it doesn't guarantee that the generator will actually satisfy them in the future frames—it only ensures the "tools" to do so remain in the cache.

Summary updated

Abstract

Agentic autoregressive video generation combines semantic planning, sequential generation, and visual verification for controllable synthesis. Growing visual histories enlarge the key-value (KV) cache, increasing memory use and attention costs across candidate rollouts. Existing compression methods primarily optimize individual generators without the planner's semantic constraints, risking low priority for states supporting requested attributes or actions. We introduce SemanticKV, a training-free method for semantically aligned KV cache compression. To our knowledge, SemanticKV is the first KV cache compression method for agentic autoregressive video generation that explicitly uses planner-extracted semantic constraints to guide retention. Text cross-attention identifies constraint-related video positions, whose self-attention queries are collected into separate groups. Attention weights are averaged within each group and maximized across groups, giving each cached entry a high score when strongly used by any represented constraint. A single top-k selection retains high-scoring entries under the memory budget, leaving their keys, values, and positions unchanged. For fixed query groups and equal entry costs, this selection minimizes an additive upper bound on worst-group discarded attention. The method operates independently within each candidate without retraining or additional video generations. We evaluate four AR video backbones on three benchmarks for 5-, 30-, and 60-second generation. On Causal Forcing at a compression ratio of 0.7, SemanticKV obtains a VBench Total Score of 84.10, compared with 84.05 for the strongest compressed baseline and 83.99 for Full KV. It achieves a 1.22x speedup over Full KV and reduces KV-cache memory from 6.04 GB to 1.81 GB, a 3.35x reduction.
BibTeX
@misc{huang2026semantically,
  author = {Weizhong Huang and Huaijia Kou and Tianyu Xie and Xiawu Zheng and Liujuan Cao},
  title = {{Semantically Aligned KV Cache Compression for Efficient Agentic Autoregressive Video Generation}},
  year = {2026},
  eprint = {2611.11452},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11452},
}
Figure