Introduction
Generative modeling has traditionally relied on a two-stage paradigm: first, compressing high-dimensional data into a compact latent space using specialized autoencoders (VAEs), and second, training a generative model within that latent space. While successful, this approach requires domain-specific inductive biases—such as convolutions for images or specialized "Evoformers" for proteins—limiting the development of a truly unified generative framework.
INRFlow proposes a "one-size-fits-all" recipe for generative modeling. By treating data as a continuous mapping from coordinates to values—known as Implicit Neural Representations (INRs)—and applying Flow Matching directly in the original "ambient" data space, INRFlow eliminates the need for complex, hand-crafted compressors.
Methodology: The INRFlow Framework
The core innovation of INRFlow lies in its domain-agnostic representation of data. Instead of viewing an image as a discrete grid of pixels, the model treats it as a function $f$ mapping coordinates $(x, y)$ to color values $(R, G, B)$. This formulation extends naturally to 3D point clouds (mapping coordinates to occupancy) and proteins (mapping sequences to 3D atomic coordinates).
Architecture and Flow Matching
INRFlow utilizes a PerceiverIO transformer architecture. To manage the computational complexity of high-resolution data, the authors introduce spatial-aware latents. Each latent token is assigned a "pseudo-coordinate" and performs cross-attention only with nearby data points, ensuring efficiency without sacrificing global context.
The model is trained using Flow Matching, which learns a deterministic velocity field to push Gaussian noise toward the data distribution along straight paths. This is complemented by a Conditionally Independent CFM (CICFM) loss, which allows the model to predict the value of any single point independently given the latents. This independence enables the model to be trained on random sub-samples and evaluated at any resolution.
Experiments and Results
The authors evaluated INRFlow across three distinct domains: 2D images, 3D point clouds, and protein structures.
Cross-Domain Success
INRFlow demonstrated competitive performance across all tested modalities:
- Images: On ImageNet-128, the model achieved an FID of 2.73, outperforming several domain-specific diffusion models like ADM. It also set new benchmarks for "function space" models on FFHQ and LSUN-Church.
- 3D Point Clouds: Using the Objaverse dataset, INRFlow successfully generated complex 3D objects from single-image prompts.
- Proteins: The model produced physically plausible protein backbones from amino-acid sequences using the SwissProt dataset.
Key Insights
A significant finding is the model's resolution agnosticism. Because it learns a continuous function, a model trained on 256x256 images can generate 2048x2048 images at inference time by sampling the learned density at a finer grain. Furthermore, the model exhibits strong scaling behavior, with sample quality improving consistently as the model size increases from 35M to 733M parameters.
Discussion
INRFlow represents a significant step toward a unified generative foundation model. By bypassing the latent space, it avoids the limitations of domain-specific compressors. However, training directly in ambient space remains computationally expensive. While sub-sampling techniques mitigate this, the total Gflops required are higher than latent-space counterparts. Future research will focus on multi-modal co-training to further leverage the domain-agnostic nature of the INRFlow architecture.
Summary updated