f-DM: A Multi-stage Diffusion Model via Progressive Signal Transformation

Jiatao Gu, Shuangfei Zhai, Yizhe Zhang, Miguel Ángel Bautista, Joshua M. Susskind

TL;DR

$f$-DM is a generalized family of diffusion models that breaks the "fixed resolution" constraint of standard diffusion. By incorporating progressive transformations—like downsampling, blurring, or VAE encoding—directly into the diffusion process, it achieves state-of-the-art image quality with roughly double the inference speed and better semantic control.

The Problem: Rigid Latent Spaces in Diffusion

Standard Diffusion Models (DMs) are powerful but computationally rigid. They typically operate in the same high-dimensional "ambient" space (e.g., 256x256 pixels) throughout the entire noising and denoising process. This is inefficient: why spend heavy compute on fine details when the image is still mostly noise?

While models like Variational Autoencoders (VAEs) and GANs naturally use a "coarse-to-fine" hierarchy, standard DMs struggle to change latent spaces mid-stream. Existing "cascaded" diffusion approaches solve this by training multiple separate models, but this introduces domain gaps and error propagation. The authors of $f$-DM sought a way to unify hierarchical transformations into a single, continuous diffusion framework.

Methods: Multi-stage Diffusion Framework

Comparison of DMs, VAEs, and f-DM

The core innovation is the Multi-stage Diffusion framework. Instead of a single process, $f$-DM splits the diffusion timeline into stages, each defined by a signal transformation $f$ (e.g., downsampling) and its inverse $g$ (e.g., upsampling).

Key technical components include:

  • Interpolation-based Formulation: To prevent "shocks" to the model when switching resolutions, the authors use a linear interpolation between the transformed signal and its approximation. This ensures the signal corrupts smoothly across stage boundaries.
  • Resolution-agnostic SNR: The authors discovered that standard Signal-to-Noise Ratio (SNR) measures fail when resolution changes because they average noise element-wise. They propose a new rescaling recipe (Signal Preserved or Variance Preserved) that keeps the "perceived noisiness" consistent across different sampling rates.
  • Double Reconstruction Loss: The model is trained to jointly predict the denoised output and the signal degradation. This "double prediction" allows the model to simultaneously remove Gaussian noise and reverse the signal transformation (like blurring or downsampling).
  • Flexible Transformations: The framework supports hand-designed functions (downsampling, Gaussian blur) and learned ones (pretrained VQVAE/VQGAN encoders).

Key Insights and Capabilities

Progressive signal transformations in f-DM
  • Noise Rescaling is Essential: Without adjusting noise levels during sub-sampling, low-resolution stages become "too noisy" too quickly, making it impossible for the model to learn meaningful global structures.
  • Bridging the VAE/DM Gap: Unlike Latent Diffusion Models (LDM), which are strictly limited by the quality of a frozen VAE decoder, $f$-DM learns diffusion in both latent and image spaces. This allows it to "fix" artifacts that a standard VAE might leave behind.
  • Zero-Shot Conditional Tasks: Because the model learns to reverse transformations like blurring and downsampling, a pre-trained unconditional $f$-DM can perform tasks like super-resolution and de-blurring without any additional training.
  • Semantic Hierarchy: The multi-stage approach naturally allocates high-level semantic information to the low-resolution stages and fine-grained textures to the high-resolution stages.

Key Results

The authors evaluated $f$-DM on several standard high-resolution (256x256) image generation benchmarks, including FFHQ, AFHQ, LSUN Church & Bedroom, and ImageNet.

  • Efficiency: $f$-DM models typically achieve a ~2x speedup in inference compared to standard DDPM/DDIM baselines because the early steps are performed in lower-dimensional spaces.
  • Quality: On the AFHQ dataset, $f$-DM-DS (downsampling) achieved an FID of 6.4, significantly outperforming the standard DDPM baseline of 9.3.
  • State-of-the-Art: On ImageNet 256x256, $f$-DM-VQGAN achieved an FID of 6.8, outperforming standard DDPM (10.9) and matching or exceeding other complex hierarchical models.
  • Robustness: Qualitative results show that $f$-DM produces fewer artifacts and better global coherence than cascaded models, which often suffer from "color shift" or "texture bleeding" between stages.

Limitations and Future Work

  • Heuristic Scheduling: The "stage schedule" (deciding exactly when to transition from 16x16 to 32x32, etc.) is currently determined by heuristics like linear or cosine schedules. Finding an automated, optimal schedule for any given transformation is an open question.
  • Fixed Transformations: In this paper, the transformations (like the VAE encoder) are pre-defined or pre-trained. A promising future direction is to jointly optimize the transformations and the diffusion model from scratch.
  • Sampling Speed: While 2x faster than standard DMs, it still relies on iterative sampling. Combining $f$-DM with "one-step" distillation methods could lead to even more dramatic performance gains.

Summary updated

Abstract

Diffusion models (DMs) have recently emerged as SoTA tools for generative modeling in various domains. Standard DMs can be viewed as an instantiation of hierarchical variational autoencoders (VAEs) where the latent variables are inferred from input-centered Gaussian distributions with fixed scales and variances. Unlike VAEs, this formulation constrains DMs from changing the latent spaces and learning abstract representations. In this work, we propose f-DM, a generalized family of DMs which allows progressive signal transformation. More precisely, we extend DMs to incorporate a set of (hand-designed or learned) transformations, where the transformed input is the mean of each diffusion step. We propose a generalized formulation and derive the corresponding de-noising objective with a modified sampling algorithm. As a demonstration, we apply f-DM in image generation tasks with a range of functions, including down-sampling, blurring, and learned transformations based on the encoder of pretrained VAEs. In addition, we identify the importance of adjusting the noise levels whenever the signal is sub-sampled and propose a simple rescaling recipe. f-DM can produce high-quality samples on standard image generation benchmarks like FFHQ, AFHQ, LSUN, and ImageNet with better efficiency and semantic interpretation.
BibTeX
@inproceedings{gu2023f,
  author = {Jiatao Gu and Shuangfei Zhai and Yizhe Zhang and Miguel Ángel Bautista and Joshua M. Susskind},
  title = {{f-DM: A Multi-stage Diffusion Model via Progressive Signal Transformation}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=iBdwKIsg4m},
}
Figure