Generative Modelling with Inverse Heat Dissipation

Severi Rissanen, Markus Heinonen, Arno Solin

While modern diffusion models—the engines behind tools like DALL-E and Midjourney—are incredibly successful, their underlying "forward process" is somewhat arbitrary. To create a starting point for generation, they typically destroy image information by adding isotropic Gaussian noise, treating every pixel as an independent entity.

However, natural images are inherently multi-scale: a "cat" is defined by large-scale shapes (ears, tail) and fine-scale textures (fur). Standard diffusion models have an implicit bias toward generating coarse features before fine ones, but they don't explicitly leverage the spatial structure of the image plane.

To address this, researchers propose a more structured alternative: the Inverse Heat Dissipation Model (IHDM). By explicitly modeling the "melting" of information from fine details to coarse shapes using the physics of heat dissipation, the model gains unique abilities like disentangling color from form and generating new samples from just a handful of training examples.

Comparison of standard diffusion (adding noise) and the Inverse Heat Dissipation Model (blurring).

The Inverse Heat Dissipation Approach

The IHDM replaces the standard noise-addition process with a physical blurring process, fundamentally changing how the model learns to generate images.

  1. The Forward Process (Blurring): Instead of adding white noise, the model runs the heat equation, a partial differential equation (PDE) that describes how heat spreads over time. In an image, this acts as a perfect low-pass filter, smoothing out fine details until only the average color remains.
  2. Frequency Domain Efficiency: Solving PDEs on a 2D grid can be computationally expensive. The authors cleverly use the Discrete Cosine Transform (DCT) to solve the heat equation in the frequency domain. This allows them to calculate any stage of the blurring process in $O(N \log N)$ time.
  3. Stochastic Reversal: To turn this into a generative model, a reverse process is defined. Because deblurring is an "ill-posed" problem (many sharp images can result in the same blurry one), a small amount of noise ($\sigma$) is added during training. This forces the model (a U-Net) to learn a distribution of possible "sharpenings" rather than a single deterministic one.
  4. The Prior: Since the end of the forward process is just a very blurry image (essentially a 3D vector of the average RGB values), the "starting point" for generation is much simpler than the high-dimensional noise used in standard diffusion.
The information melting forward process and the generative reverse sequence.

Key Insights and Capabilities

By baking a "coarse-to-fine" inductive bias directly into the model, IHDM demonstrates several emergent properties not typically seen in standard diffusion models:

  • Explicit Coarse-to-Fine Bias: By using the heat equation, the model is mathematically forced to deal with global structure (low frequencies) before it can even "see" the fine details (high frequencies).
  • Disentanglement of Color and Shape: Because the generation starts from a blurred "blob" of color, the authors found they could fix the random noise and change only the starting color. This results in the model generating the same face or object but with a different color palette—a feat difficult for standard diffusion models.
  • Few-Shot Generalization: Perhaps the most striking finding is that IHDM can learn to generate diverse images from tiny datasets. When trained on only 20 MNIST digits, IHDM generated new, plausible digit variations. In contrast, standard diffusion models either failed to produce recognizable shapes or simply memorized the 20 training images.
  • Smooth Interpolation: Moving between two points in the latent space of IHDM results in visually smooth transitions. Unlike standard diffusion, where interpolations often pass through "noisy" or nonsensical intermediate states, IHDM interpolations look like natural morphing.

Performance and Future Directions

The model was evaluated on standard benchmarks including MNIST, CIFAR-10, LSUN Churches, AFHQ, and FFHQ. It achieved competitive FID scores (a measure of image realism), such as 18.96 on CIFAR-10.

While it does not currently beat state-of-the-art specialized GANs or the most highly optimized DDPMs in raw image sharpness, it proves that PDE-based diffusion is a viable and robust generative framework. The authors note that the current architecture uses a standard U-Net designed for denoising; architectures specifically tailored for deblurring or multi-scale processing could significantly close this performance gap.

Furthermore, the underlying principle extends beyond images. While the heat equation is natural for 2D grids, using operators like the Graph Laplacian could allow for generative modeling on 3D meshes or social networks using the exact same "inverse dissipation" principle.

Summary updated

Abstract

While diffusion models have shown great success in image generation, their noise-inverting generative process does not explicitly consider the structure of images, such as their inherent multi-scale nature. Inspired by diffusion models and the empirical success of coarse-to-fine modelling, we propose a new diffusion-like model that generates images through stochastically reversing the heat equation, a PDE that locally erases fine-scale information when run over the 2D plane of the image. We interpret the solution of the forward heat equation with constant additive noise as a variational approximation in the diffusion latent variable model. Our new model shows emergent qualitative properties not seen in standard diffusion models, such as disentanglement of overall colour and shape in images. Spectral analysis on natural images highlights connections to diffusion models and reveals an implicit coarse-to-fine inductive bias in them.
BibTeX
@inproceedings{rissanen2023generative,
  author = {Severi Rissanen and Markus Heinonen and Arno Solin},
  title = {{Generative Modelling with Inverse Heat Dissipation}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=4PJUBT9f2Ol},
}
Figure