Abstract: Despite the recent visually-pleasing results achieved, the massive computational cost has been a long-standing flaw for diffusion probabilistic models (DPMs), which, in turn, greatly limits their applications on resource-limited platforms. Prior methods towards efficient DPM, however, have largely focused on accelerating the testing yet overlooked their huge complexity and sizes. In this paper, we make a dedicated attempt to lighten DPM while striving to preserve its favourable performance. We start by training a small-sized latent diffusion model (LDM) from scratch, but observe a significant fidelity drop in the synthetic images. Through a thorough assessment, we find that DPM is intrinsically biased against high-frequency generation, and learns to recover different frequency components at different time-steps. These properties make compact networks unable to represent frequency dynamics with accurate high-frequency estimation. Towards this end, we introduce a customized design for slim DPM, which we term as Spectral Diffusion (SD), for light-weight image synthesis. SD incorporates wavelet gating in its architecture to enable frequency dynamic feature extraction at every reverse step, and conducts spectrum-aware distillation to promote high-frequency recovery by inverse weighting the objective based on spectrum magnitude. Experimental results demonstrate that, SD achieves 8–18 × computational complexity reduction as compared to the latent diffusion models on a series of conditional and unconditional image generation tasks while retaining competitive image fidelity.
The Challenge of Slimming Diffusion Models
Diffusion Probabilistic Models (DPMs) have revolutionized generative AI, but their massive parameter counts and computational demands make them difficult to deploy on edge devices. While previous efforts have focused on reducing the number of sampling steps, the problem of model size (parameter count) and complexity (MACs) remains largely unaddressed.
Simply shrinking a Latent Diffusion Model (LDM)—for example, by reducing channel widths—results in a significant loss of image fidelity. These "slim" models typically produce blurry results, failing to capture the sharp edges and fine textures that characterize high-quality synthesis. The researchers identify this as a "frequency bias" problem: standard DPMs are naturally biased against high-frequency generation, and small networks lack the representational capacity to handle the dynamic shift in frequency focus required during the denoising process.
Frequency Bias in Denoising Trajectories
Through mathematical analysis, the authors demonstrate that the optimal denoiser in a DPM acts like a cascade of Wiener filters with growing bandwidths. This means the model learns to recover different frequency components at different time-steps: coarse global structures (low-frequency) are recovered at high noise levels, while fine local textures (high-frequency) are recovered at low noise levels.
Because high-frequency details are statistically "rare" compared to low-frequency patterns, compact networks tend to ignore them in favor of the majority low-frequency components. This leads to the characteristic blurriness seen in naive small-scale models.
Spectral Diffusion (SD)
To address these limitations, the authors propose Spectral Diffusion (SD), which introduces two key innovations:
- Dynamic Wavelet Gating (WG): Instead of standard resizing, SD uses Discrete Wavelet Transforms (DWT) for upsampling and downsampling. A Wavelet Gating module allows the model to dynamically attend to different frequency bands at different timesteps, adapting its architecture to the specific frequency needs of each denoising stage.
- Spectrum-Aware Distillation (SA-Distillation): To improve high-frequency recovery, the authors employ a distillation loss in the frequency domain. By applying an inverse-weighting scheme based on spectrum magnitude, the student model is forced to prioritize the rare, high-frequency details that it would otherwise overlook.
Performance and Efficiency
Spectral Diffusion achieves remarkable compression without sacrificing the quality of generated images. On the FFHQ dataset, SD reduces the parameter count from 274.1M to just 21.1M (a 13x reduction) while maintaining a competitive FID score of 10.5, significantly outperforming a naive slim model (Lite-LDM) of similar size.
Across various benchmarks including CelebA-HQ and LSUN, SD consistently achieves 8–18× reductions in computational complexity (MACs) and 2–5× speedups on both GPU and CPU platforms. These results demonstrate that frequency-aware architectural design and training objectives are essential for creating high-performance, lightweight diffusion models suitable for resource-constrained environments.
Summary updated