TL;DR
DiT.XPU is a novel inference framework that brings high-resolution Diffusion Transformers (DiTs) to mobile devices by optimizing sparse attention for the unique constraints of mobile NPUs. By combining hardware-friendly static masking with a clever "anchor-based" error compensation system, it achieves up to a 1.56× end-to-end speedup while maintaining image quality nearly identical to full dense attention.
What Problem Does This Paper Address?
Diffusion Transformers (DiTs) are the state-of-the-art for image generation, but they are notoriously slow on mobile devices. The primary bottleneck is the attention mechanism, which scales quadratically with image resolution. While "sparse attention" (skipping unimportant pixel interactions) can theoretically save time, existing methods struggle on mobile Systems-on-Chip (SoCs) for two reasons:
- Hardware Mismatch: Mobile Neural Processing Units (NPUs) are designed for predictable, "tile-based" math. Existing sparse methods often use "dynamic sparsity," which requires complex preprocessing (sorting and selecting tokens) that can take longer than the actual computation it's trying to save.
- Resource Fragmentation: Mobile SoCs have a CPU, GPU, and NPU, but they rarely work together efficiently. Moving data between these units often introduces massive overhead that erodes any performance gains.
Methods
The authors introduce DiT.XPU, which tackles these challenges through three main innovations:
- Hardware-Aligned Sparse Attention: Instead of trying to find a perfect sparse mask for every single step, DiT.XPU uses a static, tile-aligned "halo" mask. This mask is designed to match the NPU’s native 32x32 processing tiles, making the computation extremely efficient for the hardware.
- Anchor-Based Error Compensation: Static masks inevitably lose some information. To fix this, the framework uses "anchor steps." Every few denoising steps, it calculates a full dense attention output and a sparse one. It then caches the difference (the residual). For the following sparse steps, it reuses this cached residual to "correct" the sparse output, keeping the image quality high.
- Heterogeneous XPU Collaboration: During those heavy anchor steps, DiT.XPU doesn't just wait. It assigns the dense attention task to the NPU while the GPU concurrently handles the sparse calibration. They use shared memory mappings to let the processors talk to each other without slow data copying.
- Mask-Guided Dataflow: Since the sparse mask is known in advance, the system pre-plans all data movements. It uses asynchronous DMA (Direct Memory Access) to load the next set of data while the NPU is still working on the current one, effectively hiding the "data wait" time.
Datasets
The framework was evaluated using five prominent DiT models:
- Stable Diffusion 3.5 (Medium & Large)
- Z-Image & Z-Image-Turbo
- FLUX.2 [klein]
Quality was measured using standard benchmarks including GenEval (for prompt following), HPSv2.1 and ImageReward (for human preference), and FID/CLIP scores (for distributional quality and consistency).
Key Insights and Ideas
- Temporal Stability is the "Aha!" Moment: The authors discovered that the error introduced by skipping attention (the residual) changes very slowly across denoising steps. This means you don't need to calculate the "fix" at every step; you can do it once and reuse it for several steps without losing quality.
- Predictability Trumps Sparsity: On mobile NPUs, doing more math in a predictable, tile-aligned way is often faster than doing less math in an irregular, "dynamic" way.
- Fusing is Essential: By fusing layout conversion, normalization, and RoPE (Rotary Positional Embedding) directly into the projection kernels, they eliminated the "round-trip" to the CPU that usually kills mobile performance.
Key Results
- End-to-End Speedup: Achieved up to 1.56× speedup for the entire denoising process compared to the standard NPU backend (stable-diffusion.cpp) on Snapdragon 8 Gen 4.
- Kernel Performance: The attention kernels themselves were up to 11.5× faster than existing state-of-the-art sparse attention alternatives like PISA or SpargeAttn.
- Quality Preservation: In many cases, DiT.XPU actually scored higher on GenEval than full dense attention, likely due to the calibration effect of the anchor steps.
- Energy Efficiency: The NPU-based sparse kernels consumed an order of magnitude less energy than CPU-based dense execution.
Limitations and Future Work
- Fixed Anchoring: Currently, the "anchor steps" happen at a fixed interval (e.g., every 5 steps). The authors note that an input-adaptive schedule—where the model decides when it needs a calibration step based on the specific image being generated—could be even more efficient.
- Hardware Specificity: While the principles apply broadly, the current implementation is heavily optimized for the Qualcomm Hexagon NPU architecture. Porting these specific dataflow optimizations to other mobile SoCs (like Apple's Neural Engine or MediaTek's APU) would require additional engineering.
Summary updated