TL;DR
Video diffusion transformers are notoriously slow due to the quadratic computational cost of attention mechanisms. HEART (Heterogeneity-Exploiting Adaptive Refresh and Thresholding) is a training-free framework that accelerates these models by recognizing that attention heads evolve at different speeds and possess varying sensitivities to sparsification. By adaptively reusing attention masks and calibrating head-specific thresholds, HEART significantly boosts inference speed while preserving—and often improving—video quality.
What Problem Does This Paper Address?
Diffusion Transformers (DiTs) have become the backbone of modern video generation, but processing long spatiotemporal token sequences makes full attention a massive runtime bottleneck. To tackle this, researchers use "sparse attention" techniques (like top-$p$ sparsification) to dynamically drop less important attention blocks during inference.
However, existing training-free sparse attention methods suffer from a "one-size-fits-all" limitation. They treat every attention head identically by:
- Recomputing sparse attention masks for every head at every single denoising step, creating massive computational overhead.
- Applying a uniform sparsification threshold across all heads, ignoring the fact that some heads are far more sensitive to information loss than others.
The authors introduce HEART to exploit head heterogeneity, optimizing both when masks are updated and how much each head is sparsified.
Methods
HEART is a plug-and-play, training-free framework consisting of two core components:
- Temporal Mask Reuse (TMR): Instead of blindly recomputing attention masks at every step, TMR tracks a lightweight "query-key drift" metric. By mean-pooling features across tokens to minimize memory overhead, it checks if a head's internal representations have stabilized. If the drift is below a threshold ($\delta$), the head reuses its cached mask (see Figure 2). It also includes a layer-level gating mechanism to mitigate GPU kernel launch overheads when only a few heads need refreshing.
- Error-guided Budgeted Calibration (EBC): Performed offline, EBC profiles how individual heads react to different top-$p$ thresholds. Rather than measuring local attention errors, it evaluates how sparsification impacts the final denoising velocity. Crucially, it uses a 3D Fast Fourier Transform (3D-FFT) to weight errors, penalizing low-frequency artifacts more heavily since they are highly disruptive to human perception. Finally, it solves an Integer Linear Programming (ILP) problem to allocate optimal, head-specific thresholds under a global sparsity budget.
Datasets
The authors evaluate HEART using state-of-the-art open-source video generation models: Wan2.1 (1.3B and 14B variants) and HunyuanVideo (13B). For the offline calibration phase, they randomly sample a shared set of 10 prompts from the VBench benchmark dataset. Performance is measured using standard perceptual video metrics (VBench, VisionReward) and dense-output similarity metrics (PSNR, SSIM, LPIPS).
Key Insights and Ideas
- Mask Evolution Heterogeneity: Attention masks in certain heads remain highly stable across dozens of denoising steps, while others fluctuate rapidly. Fixed, uniform refresh schedules waste significant compute on the stable heads (see Figure 2).
- Sparsification Sensitivity: Different heads respond uniquely to the same top-$p$ threshold. Some heads can be aggressively sparsified with almost zero impact on the final video, while others cause massive error propagation if altered even slightly (see Figure 3).
- Frequency-Aware Importance: Not all errors in the denoising process affect video quality equally. Controlled perturbations reveal that low-temporal, low-spatial frequency errors degrade structural video fidelity significantly more than high-frequency noise, motivating EBC's frequency-weighted objective function.
Key Results
HEART consistently pushes the quality-efficiency frontier outward when integrated into advanced sparse attention baselines like XAttention and SVG2.
- Wan2.1-1.3B (720P resolution): Combining XAttention with HEART increased the end-to-end inference speedup from 1.67× to 1.85× (and attention-only speedup from 2.23× to 2.68×). Remarkably, this speedup came with an increase in video quality, raising the PSNR from 22.24 to 22.69 and improving SSIM.
- HunyuanVideo-13B: HEART raised XAttention's baseline PSNR from 26.06 to 28.92 while simultaneously bumping the end-to-end speedup from 1.27× to 1.32×.
Limitations and Future Work
- Additive Error Assumption: To keep the offline calibration tractable, EBC uses an additive surrogate model that treats head errors independently. This ignores complex, non-linear interactions where errors from different heads or layers might amplify or cancel each other out.
- Offline Profiling Step: Although HEART requires zero retraining or weight modifications, it still relies on an initial offline calibration run per model architecture. A compelling direction for future work would be developing fully online, dynamic threshold calibration mechanisms that adapt on-the-fly to arbitrary inputs.
Summary updated