Diffusion Probabilistic Fields

Peiye Zhuang, Samira Abnar, Jiatao Gu, Alexander G. Schwing, Joshua M. Susskind, Miguel Ángel Bautista

Abstract

Diffusion probabilistic models have emerged as a dominant paradigm for generative modeling across images, 3D geometry, and video. However, these models typically require domain-specific denoising architectures, often constrained by the assumption that data resides on a fixed Euclidean grid. We introduce Diffusion Probabilistic Fields (DPF), a framework that learns distributions over continuous functions defined over metric spaces. By extending diffusion models to an explicit field parametrization, DPF enables end-to-end learning without the need for latent vector representations. Our results demonstrate that a single DPF architecture can effectively model diverse modalities, including 2D images, 3D geometry, and signals on non-Euclidean manifolds.

Introduction

Most state-of-the-art diffusion models are "grid-locked." They assume data is structured as discrete Euclidean grids, such as pixels in a 2D image or voxels in a 3D volume. This structural dependency necessitates custom denoising architectures for every specific data type and fails to accommodate irregular data structures like point clouds or non-Euclidean geometries like spherical data.

To bridge this gap, researchers have previously explored "two-stage" methods that compress data into latent vectors. However, these often suffer from information loss, resulting in blurry or low-fidelity generations. Diffusion Probabilistic Fields (DPF) addresses these challenges by treating data as continuous functions—or fields—allowing for a unified generative framework that handles multiple modalities out of the box.

Methodology

Functional Representation of Data

The core innovation of DPF is the transition from discrete representations to a functional view. Instead of viewing an image as a $H \times W$ matrix, DPF defines it as a function $f$ that maps a coordinate $x \in \mathbb{R}^d$ to a signal $y$ (e.g., RGB values). This allows the model to represent data independently of a specific resolution or grid structure.

Explicit Field Parametrization

Unlike prior work that represents fields via latent vectors, DPF utilizes an explicit representation through context pairs. A field is defined by a set of coordinates and their corresponding signals. During the diffusion process, Gaussian noise is added only to the signals, while the underlying coordinates remain fixed. The model is then trained to reverse this process, effectively learning to denoise the continuous field.

The Score Field Network

To implement the denoising process, DPF employs a Score Field Network based on the PerceiverIO architecture. This transformer-based backbone is uniquely suited for this task as it can handle varying numbers of input points efficiently. The network processes "context pairs" to capture the global structure of the field and uses "query pairs" to predict the noise at specific coordinates. This approach ensures that the model remains agnostic to the underlying geometry of the data.

Experimental Evaluation

Cross-Domain Performance

The authors evaluated DPF across several distinct domains to demonstrate its versatility:

  • 2D Images: Using CelebA-HQ ($64 \times 64$) and CIFAR-10 ($32 \times 32$).
  • 3D Geometry: Modeling ShapeNet objects voxelized at $64^3$ resolution.
  • Spherical Data: Modeling signals on non-Euclidean surfaces, including spherical versions of MNIST, AFHQ (Animal Faces), and ERA5 global temperature data.

Key Results

DPF demonstrates significant improvements over existing domain-agnostic field models. On the CelebA-HQ $64 \times 64$ benchmark, DPF achieved a Fréchet Inception Distance (FID) of 13.21, substantially outperforming models like Functa (40.40) and GEM (30.42). In 3D shape generation on ShapeNet, DPF achieved a "Coverage" score of 0.419, indicating a superior ability to capture the multimodal distribution of 3D geometries compared to GAN-based or latent-based alternatives.

Resolution-Free Generation

A significant advantage of the continuous functional approach is resolution independence. DPF models trained on $64 \times 64$ images can perform zero-shot super-resolution by being queried at higher-density coordinates (e.g., $128 \times 128$) during inference, maintaining global consistency without additional training.

Discussion and Future Work

DPF represents a step toward a truly unified generative model. By shifting the handling of data geometry to the input coordinates rather than the network architecture, it provides a flexible framework for diverse data types. Furthermore, the authors draw a theoretical parallel between DPF and Sparse Gaussian Processes, suggesting that DPF acts as a deep generative version of a stochastic process.

Despite its strengths, DPF faces challenges common to diffusion models and transformers. The iterative sampling process of DDPMs remains computationally expensive, and the quadratic cost of attention mechanisms in PerceiverIO limits scaling to extremely high-resolution fields. Future research may focus on hierarchical modeling to capture multi-scale features more efficiently and accelerating the sampling process to make DPF more viable for real-time applications.

Summary updated

Abstract

Diffusion probabilistic models have quickly become a major approach for generative modeling of images, 3D geometry, video and other domains. However, to adapt diffusion generative modeling to these domains the denoising network needs to be carefully designed for each domain independently, oftentimes under the assumption that data lives in a Euclidean grid. In this paper we introduce Diffusion Probabilistic Fields (DPF), a diffusion model that can learn distributions over continuous functions defined over metric spaces, commonly known as fields. We extend the formulation of diffusion probabilistic models to deal with this field parametrization in an explicit way, enabling us to define an end-to-end learning algorithm that side-steps the requirement of representing fields with latent vectors as in previous approaches (Dupont et al., 2022a; Du et al., 2021). We empirically show that, while using the same denoising network, DPF effectively deals with different modalities like 2D images and 3D geometry, in addition to modeling distributions over fields defined on non-Euclidean metric spaces.
BibTeX
@inproceedings{zhuang2023diffusion,
  author = {Peiye Zhuang and Samira Abnar and Jiatao Gu and Alexander G. Schwing and Joshua M. Susskind and Miguel Ángel Bautista},
  title = {{Diffusion Probabilistic Fields}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=ik91mY-2GN},
}
Figure