TL;DR
This paper introduces the Progressive Diffusion Transformer (PiT), a novel architecture for diffusion models that significantly improves efficiency and performance. By analyzing the attention patterns in existing diffusion transformers, the authors propose Pseudo Shifted Window Attention (PSWA) and Progressive Coverage Channel Allocation (PCCA) to reduce redundant computations and better capture high-frequency details, leading to state-of-the-art results with reduced computational cost.
What Problem Does This Paper Address?
Diffusion Transformers (DiTs) have shown remarkable success in image generation by leveraging the transformer architecture. However, these models often employ computationally expensive, fully global self-attention mechanisms across all layers. This paper questions whether this extensive global computation is truly necessary, hypothesizing that diffusion models might not rely as heavily on long-range dependencies as initially assumed. Empirical analysis reveals that most attention layers exhibit significant redundancy in global computation, and conventional attention mechanisms suffer from a "low-frequency inertia," meaning they tend to over-emphasize low-frequency information and miss high-frequency details. This leads to a desire for more efficient transformer architectures that can capture essential information without the quadratic computational cost of global attention, while also addressing the low-frequency bias.
Methods
The core contributions of this paper are:
- Static Window Attention (SWA): To mitigate the computational redundancy of global attention, the authors propose restricting attention computations to local windows. This significantly reduces the computational overhead.
- Pseudo Shifted Window Attention (PSWA): To address the challenge of connecting information between windows and to combat the low-frequency bias, PSWA is introduced. It comprises two parallel branches:
- Static Window Attention Branch: Similar to SWA, it computes attention within local windows.
- High-Frequency Bridging Branch: This branch uses depthwise separable convolutions (DWConv) to simulate shifted window operations. It enriches high-frequency information and strengthens inter-window connections. Crucially, this bridging branch does not add significant computational cost and can even reduce it compared to standard attention.
- Progressive Coverage Channel Allocation (PCCA): This strategy aims to capture "Kth-order attention" without additional computational cost. It progressively shifts the channel allocation from the high-frequency bridging branch to the static window attention branch as the model depth increases. This allows later layers to implicitly capture higher-order relationships by leveraging features already aggregated from neighboring tokens in earlier layers.
Datasets
The paper uses the ImageNet-1K dataset for training and evaluation. They report results at resolutions of 256x256 and 512x512.
Key Insights and Ideas
- Global Attention is Overkill: The most striking insight is that diffusion models, despite using transformers, do not require full global attention in every layer. Analysis shows that most token interactions are local, suggesting a significant opportunity for computational savings.
- Low-Frequency Bias is Real: The paper provides evidence and theoretical backing (Theorem 1 in the supplementary material) that attention mechanisms inherently tend to favor low-frequency information, potentially missing crucial high-frequency details for fine-grained generation.
- Bridging the Gap: PSWA's dual-branch approach (static window + DWConv bridge) is a clever way to achieve both local attention efficiency and inter-window connectivity, while simultaneously injecting much-needed high-frequency information.
- Implicit Kth-Order Attention: PCCA is an elegant mechanism that avoids explicit complex Kth-order similarity computations. By progressively reallocating channels, it allows the static window attention to implicitly learn higher-order relationships based on features already enriched by the bridging branch.
Key Results
- Superior Performance: PiT-L achieves a 54% FID improvement over DiT-XL/2 with less computation.
- Efficiency Gains: PiT-B achieves an FID metric that is 24.1% better than DiT-XL/2 while using only 23.2% of its FLOPs.
- Improved Details: Visualizations show that PiT generates images with superior detail and texture compared to U-DiT, attributed to the high-frequency information captured by the bridging branch.
- Ablation Studies: Experiments confirm the effectiveness of PSWA and PCCA. Replacing MHSA with PSWA in DiT models leads to substantial improvements. The PCCA strategy, with a progressive increase in static window attention channels and a decrease in bridging channels, yields the best performance.
- Throughput: PiT models demonstrate significantly higher throughput compared to DiTs on the same GPUs.
Limitations and Future Work
- Kth-Order Range: While K=5 for the Kth-order similarity (DWConv kernel size) yields optimal results, the paper notes that K=2 and K=5 both achieve optimal FID. Further exploration into the optimal K value could be beneficial.
- Generalizability of PCCA: The paper proposes PCCA for progressive channel allocation. Future work could explore adaptive or more dynamic channel allocation strategies.
- Theoretical Foundation: While the paper provides a theoretical insight into the low-frequency bias of attention, further theoretical investigations into the properties of PSWA and PCCA could be valuable.
- Broader Applications: While focused on image generation, the principles of efficient attention mechanisms and Kth-order information capture could potentially be applied to other domains where transformers are used.
Summary updated