Diffusion Probabilistic Fields: Unifying Generative Modeling Across Modalities
Diffusion Probabilistic Fields (DPF) moves generative modeling away from discrete grids (like pixels or voxels) and toward continuous functions, known as fields. By representing data as explicit coordinate-signal pairs and using a single-stage training process, DPF provides a unified, architecture-agnostic framework that works across 2D images, 3D geometry, and non-Euclidean surfaces like spheres.
The Problem with Existing Approaches
Most state-of-the-art diffusion models are "domain-locked." A model designed for images usually relies on a UNet architecture optimized for 2D Euclidean grids, while a model for 3D point clouds might require a Graph Neural Network. This specialization makes it difficult to apply the same generative logic to diverse data types or data that doesn't live on a standard grid.
Previous attempts to model data as continuous functions (fields) often relied on a two-stage process: first, compressing the field into a latent vector, and then learning a distribution over those latents. This "latent field" approach is computationally heavy, prone to losing fine details during reconstruction, and often requires complex adversarial training. DPF seeks to simplify this by modeling the distribution of fields directly in a single, end-to-end stage.
Methods: A Unified Field Approach
The core innovation of DPF is its explicit field parametrization. Instead of using a latent bottleneck, DPF represents a data point (like an image) as a set of context pairs—tuples of coordinates and their corresponding signals (e.g., $(x, y)$ coordinates and RGB values).
- Forward Process: DPF diffuses the signal portion of these pairs by adding Gaussian noise over $T$ time-steps, while keeping the spatial coordinates fixed.
- Score Field Network: The authors use a "score network" $\epsilon_\theta$ that takes the noisy context pairs and a set of query coordinates as input. It predicts the noise at those query points.
- Architecture Agnosticism: Because the input is just a set of points, DPF isn't tied to a UNet. The authors primarily use PerceiverIO, a transformer-based architecture that uses cross-attention to handle large numbers of points efficiently. They also demonstrate that the model works with vanilla Transformers and MLP-Mixers.
- Single-Stage Training: Unlike prior work (Functa or GEM), DPF trains the denoising process directly on the field representation, skipping the need for an initial reconstruction/auto-encoding phase.
Key Insights and Capabilities
By treating data as a function $f: \mathcal{M} \to \mathcal{Y}$, the authors show that an image, a 3D shape, and a spherical signal are mathematically the same thing. This allows a single model architecture to generate a face or a chair just by changing the coordinate dimensions.
Furthermore, because DPF learns a continuous field, it is naturally capable of resolution-free generation. A model trained on $64 \times 64$ coordinates can be queried at $128 \times 128$ or higher during inference, producing sharper results than standard bilinear upsampling. By avoiding the "latent vector" step used in previous field models, DPF avoids the "blurriness" often seen in two-stage reconstructions, capturing global consistency and fine-grained texture simultaneously.
Key Results
The authors evaluated DPF across several modalities to prove its versatility:
- 2D Images: On CelebA-HQ ($64 \times 64$), DPF achieved an FID of 13.21, significantly outperforming other domain-agnostic field models like Functa (40.40) and GEM (30.42). It also showed higher "Precision" scores, indicating more realistic samples.
- 3D Geometry: On ShapeNet ($64^3$ resolution voxels), DPF outperformed prior field-based approaches (GASP and GEM) in terms of Coverage (0.419), suggesting it better captures the diversity of the 3D dataset.
- Non-Euclidean Success: DPF successfully generated crisp digits and animal faces on spherical geometries (Spherical MNIST and AFHQ), as well as ERA5 global temperature data, proving it isn't restricted to the "flat" Euclidean planes that standard CNNs require.
Limitations and Future Work
While PerceiverIO helps manage computational load, the attention mechanism in transformers still becomes expensive as the number of query/context points increases. Generating very high-resolution fields (e.g., $1024 \times 1024$) remains a scaling challenge. Additionally, like all standard diffusion models, DPF requires many iterative steps to generate a sample, making it slower than GANs.
Future work could explore faster sampling methods (like DDIM) and more intelligent ways to choose "inducing points" or context pairs, potentially drawing inspiration from Sparse Gaussian Processes to improve efficiency.
Summary updated