D2iT: Dynamic Diffusion Transformer for Accurate Image Generation.

Weinan Jia, Mengqi Huang, Nan Chen, Lei Zhang, Zhendong Mao

Introduction

The Diffusion Transformer (DiT) has emerged as a powerful architecture for high-fidelity image generation. However, standard DiT models employ a "one-size-fits-all" approach to image compression. Every region of an image, regardless of its visual complexity, is processed using a fixed compression ratio. This leads to a significant trade-off: large compression ratios save computation but sacrifice local detail, while small compression ratios preserve detail at the cost of quadratic computational growth and potential loss of global consistency.

Comparison of existing DiT pipelines with the proposed dynamic compression approach in D2iT.

The D2iT Framework

To address these limitations, the authors introduce the Dynamic Diffusion Transformer ($D^2iT$), which adaptively allocates computational resources based on local information density. The framework operates in two distinct stages.

Stage 1: Dynamic VAE (DVAE)

Unlike traditional VAEs that apply uniform downsampling, the DVAE utilizes a hierarchical encoder to analyze image complexity. By calculating the Shannon entropy of different regions, the model identifies high-detail areas that require less downsampling (e.g., 8x) and smooth regions that can be compressed more aggressively (e.g., 16x). This results in a multi-grained latent representation that is inherently more efficient.

Stage 2: Dynamic Diffusion Transformer ($D^2iT$)

The denoising process is handled by two specialized transformer components:

  1. Dynamic Grain Transformer: This module predicts a "grain map" from noise, identifying which regions require fine-grained attention.
  2. Dynamic Content Transformer: This component uses a dual-network architecture. A Multi-grained Net establishes global structure using coarse tokens, while a Fine-grained Net (RefineNet) adds intricate details to specific regions identified by the grain map. To maintain efficiency, the RefineNet employs "mini-windows" for local processing, avoiding the high cost of global self-attention.
Detailed architecture of the two-stage D2iT framework, including the DVAE and the Dynamic Grain/Content Transformers.

Experimental Results

The $D^2iT$ model was evaluated on the ImageNet and FFHQ benchmarks, demonstrating superior performance and efficiency.

  • Image Quality: On ImageNet (256x256), $D^2iT$-XL achieved an FID score of 1.73, a significant improvement over the standard DiT-XL's score of 2.27.
  • Unconditional Generation: On the FFHQ dataset, the model achieved an FID of 4.47, representing a 28.6% improvement over the DiT-L baseline.
  • Efficiency: $D^2iT$ achieved these results while utilizing only 57.1% of the training resources required by standard DiT models, thanks to its adaptive token processing strategy.

Conclusion

$D^2iT$ represents a shift toward content-aware computation in generative modeling. By leveraging Shannon entropy to guide dynamic compression, the model achieves a better balance between global consistency and local realism. Future work may explore extending this dynamic approach to video generation, where temporal redundancy offers even greater opportunities for optimization.

Summary updated

BibTeX
@inproceedings{jia2025d,
  author = {Weinan Jia and Mengqi Huang and Nan Chen and Lei Zhang and Zhendong Mao},
  title = {{D2iT: Dynamic Diffusion Transformer for Accurate Image Generation.}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01200},
}
Figure