Diffusion Probabilistic Fields

Peiye Zhuang, Samira Abnar, Jiatao Gu, Alex Schwing, Joshua M. Susskind, Miguel Ángel Bautista

Diffusion Probabilistic Fields: Unifying Generative Modeling Across Modalities

Diffusion Probabilistic Fields (DPF) moves generative modeling away from discrete grids (like pixels or voxels) and toward continuous functions, known as fields. By representing data as explicit coordinate-signal pairs and using a single-stage training process, DPF provides a unified, architecture-agnostic framework that works across 2D images, 3D geometry, and non-Euclidean surfaces like spheres.

The Problem with Existing Approaches

Most state-of-the-art diffusion models are "domain-locked." A model designed for images usually relies on a UNet architecture optimized for 2D Euclidean grids, while a model for 3D point clouds might require a Graph Neural Network. This specialization makes it difficult to apply the same generative logic to diverse data types or data that doesn't live on a standard grid.

Previous latent field methods require a complex two-stage process: reconstruction and latent distribution learning.

Previous attempts to model data as continuous functions (fields) often relied on a two-stage process: first, compressing the field into a latent vector, and then learning a distribution over those latents. This "latent field" approach is computationally heavy, prone to losing fine details during reconstruction, and often requires complex adversarial training. DPF seeks to simplify this by modeling the distribution of fields directly in a single, end-to-end stage.

Methods: A Unified Field Approach

The core innovation of DPF is its explicit field parametrization. Instead of using a latent bottleneck, DPF represents a data point (like an image) as a set of context pairs—tuples of coordinates and their corresponding signals (e.g., $(x, y)$ coordinates and RGB values).

DPF unifies diverse modalities (2D, 3D, spherical) as continuous fields and applies a forward diffusion process directly to the signal.
  • Forward Process: DPF diffuses the signal portion of these pairs by adding Gaussian noise over $T$ time-steps, while keeping the spatial coordinates fixed.
  • Score Field Network: The authors use a "score network" $\epsilon_\theta$ that takes the noisy context pairs and a set of query coordinates as input. It predicts the noise at those query points.
  • Architecture Agnosticism: Because the input is just a set of points, DPF isn't tied to a UNet. The authors primarily use PerceiverIO, a transformer-based architecture that uses cross-attention to handle large numbers of points efficiently. They also demonstrate that the model works with vanilla Transformers and MLP-Mixers.
  • Single-Stage Training: Unlike prior work (Functa or GEM), DPF trains the denoising process directly on the field representation, skipping the need for an initial reconstruction/auto-encoding phase.

Key Insights and Capabilities

By treating data as a function $f: \mathcal{M} \to \mathcal{Y}$, the authors show that an image, a 3D shape, and a spherical signal are mathematically the same thing. This allows a single model architecture to generate a face or a chair just by changing the coordinate dimensions.

Furthermore, because DPF learns a continuous field, it is naturally capable of resolution-free generation. A model trained on $64 \times 64$ coordinates can be queried at $128 \times 128$ or higher during inference, producing sharper results than standard bilinear upsampling. By avoiding the "latent vector" step used in previous field models, DPF avoids the "blurriness" often seen in two-stage reconstructions, capturing global consistency and fine-grained texture simultaneously.

Key Results

The authors evaluated DPF across several modalities to prove its versatility:

  • 2D Images: On CelebA-HQ ($64 \times 64$), DPF achieved an FID of 13.21, significantly outperforming other domain-agnostic field models like Functa (40.40) and GEM (30.42). It also showed higher "Precision" scores, indicating more realistic samples.
  • 3D Geometry: On ShapeNet ($64^3$ resolution voxels), DPF outperformed prior field-based approaches (GASP and GEM) in terms of Coverage (0.419), suggesting it better captures the diversity of the 3D dataset.
  • Non-Euclidean Success: DPF successfully generated crisp digits and animal faces on spherical geometries (Spherical MNIST and AFHQ), as well as ERA5 global temperature data, proving it isn't restricted to the "flat" Euclidean planes that standard CNNs require.

Limitations and Future Work

While PerceiverIO helps manage computational load, the attention mechanism in transformers still becomes expensive as the number of query/context points increases. Generating very high-resolution fields (e.g., $1024 \times 1024$) remains a scaling challenge. Additionally, like all standard diffusion models, DPF requires many iterative steps to generate a sample, making it slower than GANs.

Future work could explore faster sampling methods (like DDIM) and more intelligent ways to choose "inducing points" or context pairs, potentially drawing inspiration from Sparse Gaussian Processes to improve efficiency.

Summary updated

Abstract

Diffusion probabilistic models have quickly become a major approach for generative modeling of images, 3D geometry, video and other domains. However, to adapt diffusion generative modeling to these domains the denoising network needs to be carefully designed for each domain independently, oftentimes under the assumption that data lives in a Euclidean grid. In this paper we introduce Diffusion Probabilistic Fields (DPF), a diffusion model that can learn distributions over continuous functions defined over metric spaces, commonly known as fields. We extend the formulation of diffusion probabilistic models to deal with this field parametrization in an explicit way, enabling us to define an end-to-end learning algorithm that side-steps the requirement of representing fields with latent vectors as in previous approaches (Dupont et al., 2022a; Du et al., 2021). We empirically show that, while using the same denoising network, DPF effectively deals with different modalities like 2D images and 3D geometry, in addition to modeling distributions over fields defined on non-Euclidean metric spaces.
BibTeX
@inproceedings{zhuang2023diffusion,
  author = {Peiye Zhuang and Samira Abnar and Jiatao Gu and Alex Schwing and Joshua M. Susskind and Miguel Ángel Bautista},
  title = {{Diffusion Probabilistic Fields}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=ik91mY-2GN},
}
Figure