Abstract
Diffusion models (DMs) have recently emerged as SoTA tools for generative modeling in various domains. Standard DMs can be viewed as an instantiation of hierarchical variational autoencoders (VAEs) where the latent variables are inferred from input-centered Gaussian distributions with fixed scales and variances. Unlike VAEs, this formulation limits DMs from changing the latent spaces and learning abstract representations. In this work, we propose f-DM, a generalized family of DMs which allows progressive signal transformation. More precisely, we extend DMs to incorporate a set of (hand-designed or learned) transformations, where the transformed input is the mean of each diffusion step. We propose a generalized formulation and derive the corresponding de-noising objective with a modified sampling algorithm. As a demonstration, we apply f-DM in image generation tasks with a range of functions, including down-sampling, blurring, and learned transformations based on the encoder of pretrained VAEs. In addition, we identify the importance of adjusting the noise levels whenever the signal is sub-sampled and propose a simple rescaling recipe. f-DM can produce high-quality samples on standard image generation benchmarks like FFHQ, AFHQ, LSUN, and ImageNet with better efficiency and semantic interpretation.
Introduction
Standard Diffusion Models (DMs) have revolutionized generative AI, yet they face a fundamental limitation: they are "resolution-locked." By operating at a fixed, high-dimensional resolution throughout the entire denoising process, these models incur high computational costs and struggle to capture the hierarchical abstractions that models like Variational Autoencoders (VAEs) naturally learn. In "f-DM: A Multi-stage Diffusion Model via Progressive Signal Transformation," researchers introduce a framework that allows the signal itself to transform—shrinking, blurring, or moving into latent spaces—as the diffusion process unfolds.
The f-DM Framework
The core innovation of f-DM is the integration of signal transformations directly into the generative process. Instead of diffusing toward a static target image, the model diffuses toward a progressively evolving signal. This approach bridges the gap between the high-fidelity generation of DMs and the hierarchical structure of VAEs.
As illustrated in the figure above, f-DM generalizes the diffusion process. While standard DMs (a) maintain a constant resolution and VAEs (b) utilize a fixed hierarchy of latents, f-DM (c) allows for a dynamic transition between latent and image spaces. This multi-stage architecture enables the model to focus on global semantic features in early stages before refining local details.
Progressive Signal Transformation
The f-DM process is typically divided into stages. In the initial "Latent space diffusion" stage, the model operates on a coarse representation, such as a downsampled or blurred version of the target. As the denoising progresses, the signal "unfolds" into the full-resolution image space.
This transition is managed through an interpolation-based formulation to prevent artifacts during resolution jumps. Furthermore, the authors address a critical challenge in multi-resolution diffusion: the Signal-to-Noise Ratio (SNR). When a signal is downsampled, noise tends to average out, artificially increasing the SNR. f-DM employs a resolution-agnostic rescaling recipe to maintain consistent noise levels across all stages.
Key Results and Efficiency
By shifting a significant portion of the sampling process to lower resolutions, f-DM achieves a ~2x speedup in inference compared to standard DDPMs. Beyond efficiency, the model improves generation quality; for instance, achieving an FID of 6.4 on the AFHQ dataset, a notable improvement over the 9.3 baseline.
The hierarchical nature of f-DM also provides unique semantic control. Since the early stages govern global structure and the later stages handle fine textures, users can perform latent space manipulation—such as changing a subject's expression—by only modifying the noise in specific high-resolution stages. Additionally, the model's inherent understanding of transformations allows it to perform tasks like super-resolution and de-blurring in a zero-shot manner.
Conclusion
f-DM represents a significant step forward in making diffusion models more efficient and semantically interpretable. By incorporating progressive signal transformations, it successfully combines the generative prowess of DMs with the hierarchical elegance of VAEs, setting a new standard for high-resolution image synthesis.
Summary updated