Scale-DiT: Ultra-High-Resolution Image Generation with Hierarchical Local Attention

Yuyao Zhang, Yu-Wing Tai

Abstract

Ultra-high-resolution text-to-image generation demands both fine-grained texture synthesis and globally coherent structure, yet current diffusion models remain constrained to sub-$1K \times 1K$ resolutions due to the prohibitive quadratic complexity of attention and the scarcity of native $4K$ training data. We present Scale-DiT, a new diffusion framework that introduces hierarchical local attention with low-resolution global guidance, enabling efficient, scalable, and semantically coherent image synthesis at ultra-high resolutions. Specifically, high-resolution latents are divided into fixed-size local windows to reduce attention complexity from quadratic to near-linear, while a low-resolution latent equipped with scaled positional anchors injects global semantics. A lightweight LoRA adaptation bridges global and local pathways during denoising, ensuring consistency across structure and detail. To maximize inference efficiency, we repermute token sequence in Hilbert curve order and implement a fused-kernel for skipping masked operations, resulting in a GPU-friendly design. Extensive experiments demonstrate that Scale-DiT achieves more than $2\times$ faster inference and lower memory usage compared to dense attention baselines, while reliably scaling to $4K \times 4K$ resolution without requiring additional high-resolution training data. On both quantitative benchmarks (FID, IS, CLIP Score) and qualitative comparisons, Scale-DiT delivers superior global coherence and sharper local detail, matching or outperforming state-of-the-art methods that rely on native 4K training.

Introduction

While text-to-image (T2I) models like Stable Diffusion and FLUX have revolutionized AI art, they are largely restricted to $1K$ resolutions. Pushing to $4K$ and beyond presents two massive hurdles: the Quadratic Bottleneck, where standard attention mechanisms grow quadratically ($O(N^2)$) with pixel count, and Data Scarcity, as high-quality, captioned $4K$ datasets are rare. Most models trained on $1K$ data fail to maintain global structure—often generating artifacts like "double heads"—when forced to generate at higher resolutions.

Methodology

Scale-DiT rethinks the generation process through a hierarchical lens, focusing on local details while maintaining global composition. The framework introduces three core innovations:

  1. Hierarchical Local Attention: Instead of global pixel-to-pixel attention, Scale-DiT partitions high-resolution images into fixed-size local windows (e.g., $256\times256$). This restricts attention to these windows, transforming quadratic costs into near-linear complexity.
  2. Low-Resolution Global Guidance: To ensure local windows remain semantically connected, the model generates a low-resolution version of the image simultaneously. This "guide" uses scaled positional anchors to inject global context into every high-resolution window.
  3. Hilbert Curve Permutation & Efficiency: To optimize for GPUs, image tokens are re-ordered using a Hilbert curve. This ensures tokens in the same local window are stored contiguously in memory, enabling efficient block-sparse attention kernels that skip unnecessary computations.
Overview of the Scale-DiT framework, illustrating the hierarchical local attention mechanism, Hilbert-order permutation for efficiency, and low-resolution global guidance.

The system utilizes LoRA (Low-Rank Adaptation) to bridge these global and local pathways. Crucially, this adaptation is trained only on standard $1024\times1024$ data, yet it generalizes zero-shot to $4K$ resolutions.

Experimental Setup

Scale-DiT is built upon the FLUX.1-dev backbone. Fine-tuning was conducted on a modest dataset of 10,000 images at $1024\times1024$ resolution. Evaluation utilized a benchmark of 1,000 diverse prompts compared against the LAION-High-Resolution dataset using standard metrics including FID, Inception Score (IS), and CLIP Score.

Results and Discussion

Scale-DiT achieves a >2× speedup in inference time and significant memory savings, requiring only 35.8 GB of VRAM at $4K$ compared to 47.2 GB for dense baselines. This allows $4K$ generation on consumer-grade hardware like a 40 GB A100.

Qualitative comparison of Scale-DiT against state-of-the-art high-resolution generation methods across various prompts.

Quantitatively, Scale-DiT outperforms or matches state-of-the-art models like SANA, HiFlow, and I-MAX. Qualitatively, the model avoids common artifacts like tiling boundaries or blurry textures, producing sharp details—such as individual fur strands or complex iris structures—that remain globally coherent. The framework demonstrates Resolution Invariance, where the representational quality remains constant because local windows operate within sizes the model has already mastered during training.

Conclusion

Scale-DiT demonstrates that $4K$ image generation does not require native $4K$ training data. By aligning low-resolution global guides with high-resolution local windows via positional anchors, the model learns a scalable relationship between detail and structure. Future work will explore recursive scaling for $8K$ generation and improving global coherence for extremely complex multi-object prompts.

Summary updated

Abstract

Ultra-high-resolution text-to-image generation demands both fine-grained texture synthesis and globally coherent structure, yet current diffusion models remain constrained to sub-$1K \times 1K$ resolutions due to the prohibitive quadratic complexity of attention and the scarcity of native $4K$ training data. We present \textbf{Scale-DiT}, a new diffusion framework that introduces hierarchical local attention with low-resolution global guidance, enabling efficient, scalable, and semantically coherent image synthesis at ultra-high resolutions. Specifically, high-resolution latents are divided into fixed-size local windows to reduce attention complexity from quadratic to near-linear, while a low-resolution latent equipped with scaled positional anchors injects global semantics. A lightweight LoRA adaptation bridges global and local pathways during denoising, ensuring consistency across structure and detail. To maximize inference efficiency, we repermute token sequence in Hilbert curve order and implement a fused-kernel for skipping masked operations, resulting in a GPU-friendly design. Extensive experiments demonstrate that Scale-DiT achieves more than $2\times$ faster inference and lower memory usage compared to dense attention baselines, while reliably scaling to $4K \times 4K$ resolution without requiring additional high-resolution training data. On both quantitative benchmarks (FID, IS, CLIP Score) and qualitative comparisons, Scale-DiT delivers superior global coherence and sharper local detail, matching or outperforming state-of-the-art methods that rely on native 4K training. Taken together, these results highlight hierarchical local attention with guided low-resolution anchors as a promising and effective approach for advancing ultra-high-resolution image generation.
BibTeX
@misc{zhang2025scale,
  author = {Yuyao Zhang and Yu-Wing Tai},
  title = {{Scale-DiT: Ultra-High-Resolution Image Generation with Hierarchical Local Attention}},
  year = {2025},
  eprint = {2510.16325},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.16325},
}
Figure