Real-Time Video Generation with Pyramid Attention Broadcast

Xuanlei Zhao, Xiaolong Jin, Kai Wang, Yang You

Abstract

We present Pyramid Attention Broadcast (PAB), a real-time, high-quality, and training-free approach for Diffusion Transformer (DiT)-based video generation. Our method is founded on the observation that attention difference in the diffusion process exhibits a U-shaped pattern, indicating significant redundancy. We mitigate this by broadcasting attention outputs to subsequent steps in a pyramid style, applying different broadcast strategies to each attention type based on their variance. We further introduce broadcast sequence parallel for more efficient distributed inference. PAB demonstrates superior results across three models, achieving real-time generation for up to 720p videos.

Introduction

High-quality video generation models, such as Sora, have significantly advanced the field of generative AI. However, these models rely on Diffusion Transformers (DiTs), which are computationally expensive and memory-intensive. While model compression techniques like distillation or pruning can mitigate these costs, they often require extensive re-training and large datasets. Existing training-free caching methods, primarily designed for image generation, frequently fail when applied to video DiTs due to the complex interplay of spatial, temporal, and cross-attention mechanisms.

Baseline performance of original DiT models on a single GPU.

Methodology

Pyramid Attention Broadcast (PAB) addresses these challenges by exploiting temporal redundancies in the diffusion process without requiring additional training. The approach is built on three core pillars:

1. The U-Shaped Redundancy Pattern

Empirical analysis reveals that the difference in attention outputs between consecutive diffusion steps follows a U-shaped curve. While the initial and final steps involve rapid changes, the middle ~70% of the process remains remarkably stable. PAB leverages this stability by reusing attention outputs across these redundant steps.

2. Pyramid Broadcasting Strategy

PAB employs a hierarchical approach to attention reuse. Based on the variance observed in different attention types, it assigns specific "broadcast ranges":

  • Cross-Attention: The most stable component, anchoring the video to the text prompt, allows for the longest reuse range.
  • Temporal Attention: Moderately stable, capturing motion across frames.
  • Spatial Attention: The most dynamic component, requiring more frequent updates to maintain fine-grained pixel quality.

3. Broadcast Sequence Parallel (BSP)

In multi-GPU environments, communication overhead often limits scaling. PAB introduces Broadcast Sequence Parallel, which eliminates the need for heavy communication during temporal attention steps by reusing previously computed outputs. This leads to near-linear performance scaling as more GPUs are added.

Evaluation and Results

The effectiveness of PAB was evaluated using three state-of-the-art DiT models: Open-Sora, Open-Sora-Plan, and Latte. PAB achieves significant performance gains while maintaining high visual fidelity. On a cluster of 8 NVIDIA H100 GPUs, PAB achieved a 10.5× speedup for the Open-Sora model.

PAB achieves 21.3 FPS and a 10.5x speedup using 8 GPUs, enabling real-time generation.

Key results include:

  • Real-Time Generation: The system reached 21.3 FPS for 480p video and enabled real-time 720p generation.
  • Quality Preservation: On the VBench benchmark, PAB-equipped models scored nearly identically to the original versions (e.g., 78.51 vs 79.22 for Open-Sora), indicating negligible quality loss.
  • Training-Free Integration: The method works "out of the box" on existing pre-trained models, making it immediately accessible for deployment.

Conclusion

Pyramid Attention Broadcast provides a robust, training-free baseline for accelerating video generation. By identifying and exploiting the U-shaped redundancy in attention mechanisms, PAB enables real-time performance on modern hardware. Future research may explore adaptive broadcast ranges and extending these optimization principles to other transformer components like MLP layers.

Summary updated

Abstract

We present Pyramid Attention Broadcast (PAB), a real-time, high quality and training-free approach for DiT-based video generation. Our method is founded on the observation that attention difference in the diffusion process exhibits a U-shaped pattern, indicating significant redundancy. We mitigate this by broadcasting attention outputs to subsequent steps in a pyramid style. It applies different broadcast strategies to each attention based on their variance for best efficiency. We further introduce broadcast sequence parallel for more efficient distributed inference. PAB demonstrates superior results across three models compared to baselines, achieving real-time generation for up to 720p videos. We anticipate that our simple yet effective method will serve as a robust baseline and facilitate future research and application for video generation.
BibTeX
@inproceedings{zhao2025real,
  author = {Xuanlei Zhao and Xiaolong Jin and Kai Wang and Yang You},
  title = {{Real-Time Video Generation with Pyramid Attention Broadcast}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=hDBrQ4DApF},
}
Figure