HEART: Exploiting Head Heterogeneity in Sparse Attention for Video Diffusion

Xuzhe Zheng, Yuexiao Ma, Jing Xu, Xiawu Zheng, Rongrong Ji, Fei ChaoarXiv 2026 · arXiv

TL;DR

Video diffusion transformers are notoriously slow due to the quadratic computational cost of attention mechanisms. HEART (Heterogeneity-Exploiting Adaptive Refresh and Thresholding) is a training-free framework that accelerates these models by recognizing that attention heads evolve at different speeds and possess varying sensitivities to sparsification. By adaptively reusing attention masks and calibrating head-specific thresholds, HEART significantly boosts inference speed while preserving—and often improving—video quality.

What Problem Does This Paper Address?

Diffusion Transformers (DiTs) have become the backbone of modern video generation, but processing long spatiotemporal token sequences makes full attention a massive runtime bottleneck. To tackle this, researchers use "sparse attention" techniques (like top-$p$ sparsification) to dynamically drop less important attention blocks during inference.

However, existing training-free sparse attention methods suffer from a "one-size-fits-all" limitation. They treat every attention head identically by:

  1. Recomputing sparse attention masks for every head at every single denoising step, creating massive computational overhead.
  2. Applying a uniform sparsification threshold across all heads, ignoring the fact that some heads are far more sensitive to information loss than others.

The authors introduce HEART to exploit head heterogeneity, optimizing both when masks are updated and how much each head is sparsified.

Methods

HEART is a plug-and-play, training-free framework consisting of two core components:

  1. Temporal Mask Reuse (TMR): Instead of blindly recomputing attention masks at every step, TMR tracks a lightweight "query-key drift" metric. By mean-pooling features across tokens to minimize memory overhead, it checks if a head's internal representations have stabilized. If the drift is below a threshold ($\delta$), the head reuses its cached mask (see Figure 2). It also includes a layer-level gating mechanism to mitigate GPU kernel launch overheads when only a few heads need refreshing.
  2. Error-guided Budgeted Calibration (EBC): Performed offline, EBC profiles how individual heads react to different top-$p$ thresholds. Rather than measuring local attention errors, it evaluates how sparsification impacts the final denoising velocity. Crucially, it uses a 3D Fast Fourier Transform (3D-FFT) to weight errors, penalizing low-frequency artifacts more heavily since they are highly disruptive to human perception. Finally, it solves an Integer Linear Programming (ILP) problem to allocate optimal, head-specific thresholds under a global sparsity budget.

Datasets

The authors evaluate HEART using state-of-the-art open-source video generation models: Wan2.1 (1.3B and 14B variants) and HunyuanVideo (13B). For the offline calibration phase, they randomly sample a shared set of 10 prompts from the VBench benchmark dataset. Performance is measured using standard perceptual video metrics (VBench, VisionReward) and dense-output similarity metrics (PSNR, SSIM, LPIPS).

Key Insights and Ideas

  • Mask Evolution Heterogeneity: Attention masks in certain heads remain highly stable across dozens of denoising steps, while others fluctuate rapidly. Fixed, uniform refresh schedules waste significant compute on the stable heads (see Figure 2).
Figure 2. Temporal mask similarity for heads from a randomly selected layer, measured by adjacent-step mask IoU. Different heads follow noticeably different temporal trajectories, indicating that cross-step mask persistence varies substantially across heads.
  • Sparsification Sensitivity: Different heads respond uniquely to the same top-$p$ threshold. Some heads can be aggressively sparsified with almost zero impact on the final video, while others cause massive error propagation if altered even slightly (see Figure 3).
Figure 3. Head-wise effects of top-p thresholds under XAttention on Wan2.1-1.3B at 480P. Each curve represents one head: left, sparse-attention-induced attention-output error; middle, induced denoising-velocity error; right, realized sparsity. The diverse head-wise responses motivate head-specific threshold calibration.
  • Frequency-Aware Importance: Not all errors in the denoising process affect video quality equally. Controlled perturbations reveal that low-temporal, low-spatial frequency errors degrade structural video fidelity significantly more than high-frequency noise, motivating EBC's frequency-weighted objective function.

Key Results

HEART consistently pushes the quality-efficiency frontier outward when integrated into advanced sparse attention baselines like XAttention and SVG2.

  • Wan2.1-1.3B (720P resolution): Combining XAttention with HEART increased the end-to-end inference speedup from 1.67× to 1.85× (and attention-only speedup from 2.23× to 2.68×). Remarkably, this speedup came with an increase in video quality, raising the PSNR from 22.24 to 22.69 and improving SSIM.
  • HunyuanVideo-13B: HEART raised XAttention's baseline PSNR from 26.06 to 28.92 while simultaneously bumping the end-to-end speedup from 1.27× to 1.32×.
Figure 1. Qualitative comparison on Wan2.1-14B text-to-video generation at 480P. We compare dense attention with two representative sparse-attention baselines, XAttention [24] and SVG2 [25], and their variants enhanced by our method. These results show that our method better preserves the video quality while further improving inference efficiency.

Limitations and Future Work

  • Additive Error Assumption: To keep the offline calibration tractable, EBC uses an additive surrogate model that treats head errors independently. This ignores complex, non-linear interactions where errors from different heads or layers might amplify or cancel each other out.
  • Offline Profiling Step: Although HEART requires zero retraining or weight modifications, it still relies on an initial offline calibration run per model architecture. A compelling direction for future work would be developing fully online, dynamic threshold calibration mechanisms that adapt on-the-fly to arbitrary inputs.

Summary updated

Abstract

Sparse attention accelerates video diffusion by allowing each attention head to focus on only a small subset of interactions. Existing methods already construct head-specific sparse patterns conditioned on the input. However, we find that these heads also differ in two less obvious but practically important ways. First, for some heads, sparse attention masks remain stable across many denoising steps, whereas others change rapidly. Second, heads differ substantially in their sensitivity to sparsification: applying the same threshold can induce markedly different errors in the final denoising velocity. Ignoring these differences leads to redundant mask prediction and suboptimal threshold calibration across heads under a global sparsity budget. We present HEART, short for Heterogeneity-Exploiting Adaptive Refresh and Thresholding, a training-free framework that exploits both forms of head heterogeneity. First, Temporal Mask Reuse (TMR) uses a lightweight per-head query-key drift signal to determine whether a cached sparse mask remains reliable across denoising steps, refreshing it only when the drift exceeds a prescribed threshold. Second, Error-guided Budgeted Calibration (EBC) evaluates candidate thresholds offline using a frequency-weighted denoising-velocity error, and assigns each head an appropriate threshold under a global sparsity budget. HEART requires no retraining, weight modification, or sparse-kernel changes, and can be integrated directly into existing sparse-attention pipelines. Across Wan2.1-1.3B, Wan2.1-14B, and HunyuanVideo-13B, HEART consistently pushes the quality--efficiency frontier of advanced sparse attention methods such as XAttention and SVG2 outward.
BibTeX
@misc{zheng2026heart,
  author = {Xuzhe Zheng and Yuexiao Ma and Jing Xu and Xiawu Zheng and Rongrong Ji and Fei Chao},
  title = {{HEART: Exploiting Head Heterogeneity in Sparse Attention for Video Diffusion}},
  year = {2026},
  eprint = {2605.14513},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.14513},
}

Loading related content...

Figure