Introduction
The Diffusion Transformer (DiT) has emerged as a powerful architecture for high-fidelity image generation. However, standard DiT models employ a "one-size-fits-all" approach to image compression. Every region of an image, regardless of its visual complexity, is processed using a fixed compression ratio. This leads to a significant trade-off: large compression ratios save computation but sacrifice local detail, while small compression ratios preserve detail at the cost of quadratic computational growth and potential loss of global consistency.
The D2iT Framework
To address these limitations, the authors introduce the Dynamic Diffusion Transformer ($D^2iT$), which adaptively allocates computational resources based on local information density. The framework operates in two distinct stages.
Stage 1: Dynamic VAE (DVAE)
Unlike traditional VAEs that apply uniform downsampling, the DVAE utilizes a hierarchical encoder to analyze image complexity. By calculating the Shannon entropy of different regions, the model identifies high-detail areas that require less downsampling (e.g., 8x) and smooth regions that can be compressed more aggressively (e.g., 16x). This results in a multi-grained latent representation that is inherently more efficient.
Stage 2: Dynamic Diffusion Transformer ($D^2iT$)
The denoising process is handled by two specialized transformer components:
- Dynamic Grain Transformer: This module predicts a "grain map" from noise, identifying which regions require fine-grained attention.
- Dynamic Content Transformer: This component uses a dual-network architecture. A Multi-grained Net establishes global structure using coarse tokens, while a Fine-grained Net (RefineNet) adds intricate details to specific regions identified by the grain map. To maintain efficiency, the RefineNet employs "mini-windows" for local processing, avoiding the high cost of global self-attention.
Experimental Results
The $D^2iT$ model was evaluated on the ImageNet and FFHQ benchmarks, demonstrating superior performance and efficiency.
- Image Quality: On ImageNet (256x256), $D^2iT$-XL achieved an FID score of 1.73, a significant improvement over the standard DiT-XL's score of 2.27.
- Unconditional Generation: On the FFHQ dataset, the model achieved an FID of 4.47, representing a 28.6% improvement over the DiT-L baseline.
- Efficiency: $D^2iT$ achieved these results while utilizing only 57.1% of the training resources required by standard DiT models, thanks to its adaptive token processing strategy.
Conclusion
$D^2iT$ represents a shift toward content-aware computation in generative modeling. By leveraging Shannon entropy to guide dynamic compression, the model achieves a better balance between global consistency and local realism. Future work may explore extending this dynamic approach to video generation, where temporal redundancy offers even greater opportunities for optimization.
Summary updated