Abstract
We present Pyramid Attention Broadcast (PAB), a real-time, high-quality, and training-free approach for Diffusion Transformer (DiT)-based video generation. Our method is founded on the observation that attention difference in the diffusion process exhibits a U-shaped pattern, indicating significant redundancy. We mitigate this by broadcasting attention outputs to subsequent steps in a pyramid style, applying different broadcast strategies to each attention type based on their variance. We further introduce broadcast sequence parallel for more efficient distributed inference. PAB demonstrates superior results across three models, achieving real-time generation for up to 720p videos.
Introduction
High-quality video generation models, such as Sora, have significantly advanced the field of generative AI. However, these models rely on Diffusion Transformers (DiTs), which are computationally expensive and memory-intensive. While model compression techniques like distillation or pruning can mitigate these costs, they often require extensive re-training and large datasets. Existing training-free caching methods, primarily designed for image generation, frequently fail when applied to video DiTs due to the complex interplay of spatial, temporal, and cross-attention mechanisms.
Methodology
Pyramid Attention Broadcast (PAB) addresses these challenges by exploiting temporal redundancies in the diffusion process without requiring additional training. The approach is built on three core pillars:
1. The U-Shaped Redundancy Pattern
Empirical analysis reveals that the difference in attention outputs between consecutive diffusion steps follows a U-shaped curve. While the initial and final steps involve rapid changes, the middle ~70% of the process remains remarkably stable. PAB leverages this stability by reusing attention outputs across these redundant steps.
2. Pyramid Broadcasting Strategy
PAB employs a hierarchical approach to attention reuse. Based on the variance observed in different attention types, it assigns specific "broadcast ranges":
- Cross-Attention: The most stable component, anchoring the video to the text prompt, allows for the longest reuse range.
- Temporal Attention: Moderately stable, capturing motion across frames.
- Spatial Attention: The most dynamic component, requiring more frequent updates to maintain fine-grained pixel quality.
3. Broadcast Sequence Parallel (BSP)
In multi-GPU environments, communication overhead often limits scaling. PAB introduces Broadcast Sequence Parallel, which eliminates the need for heavy communication during temporal attention steps by reusing previously computed outputs. This leads to near-linear performance scaling as more GPUs are added.
Evaluation and Results
The effectiveness of PAB was evaluated using three state-of-the-art DiT models: Open-Sora, Open-Sora-Plan, and Latte. PAB achieves significant performance gains while maintaining high visual fidelity. On a cluster of 8 NVIDIA H100 GPUs, PAB achieved a 10.5× speedup for the Open-Sora model.
Key results include:
- Real-Time Generation: The system reached 21.3 FPS for 480p video and enabled real-time 720p generation.
- Quality Preservation: On the VBench benchmark, PAB-equipped models scored nearly identically to the original versions (e.g., 78.51 vs 79.22 for Open-Sora), indicating negligible quality loss.
- Training-Free Integration: The method works "out of the box" on existing pre-trained models, making it immediately accessible for deployment.
Conclusion
Pyramid Attention Broadcast provides a robust, training-free baseline for accelerating video generation. By identifying and exploiting the U-shaped redundancy in attention mechanisms, PAB enables real-time performance on modern hardware. Future research may explore adaptive broadcast ranges and extending these optimization principles to other transformer components like MLP layers.
Summary updated