Overview
INRFlow is a domain-agnostic generative framework that learns to create images, 3D point clouds, and protein structures using a single, unified training recipe. By applying flow matching directly in "ambient space" (the original data coordinates) rather than relying on a compressed latent space, INRFlow achieves resolution-agnostic generation and competitive performance across wildly different data types.
The Problem with Two-Stage Generation
Most state-of-the-art generative models, such as Stable Diffusion or DiT, follow a two-stage paradigm. First, they train a domain-specific compressor—like a Variational Autoencoder (VAE) for images or a PointNet for 3D data—to map raw data into a lower-dimensional latent space. Second, they train a generative model within that latent space.
While highly successful, this approach has two major drawbacks:
- Domain Dependency: Researchers must design and train a different, hand-crafted architecture for every new data type (e.g., convolutions for images, graph layers for molecules).
- Resolution Bottlenecks: Latent spaces are typically fixed-resolution, making it difficult to generate data at scales or dimensions different from the training set.
The authors of INRFlow ask a fundamental question: Can we build a single, domain-agnostic architecture that works directly on the raw data (ambient space) and handles any modality?
The INRFlow Approach: Ambient Space and INRs
To solve these bottlenecks, INRFlow draws inspiration from Implicit Neural Representations (INRs)—the concept that data can be represented as a continuous function mapping coordinates (like $x, y$ pixels) to values (like RGB colors).
Key technical components of the framework include:
- Flow Matching in Ambient Space: Instead of running diffusion in a latent space, the model learns a vector field that "flows" noise into data directly in the coordinate space.
- PerceiverIO Backbone: The framework utilizes a transformer-based architecture (PerceiverIO) that is naturally agnostic to the structural biases of the input.
- Spatial-Aware Latents: To help the transformer understand spatial relationships without relying on domain-specific biases like convolutions, INRFlow assigns "pseudo-coordinates" to latent tokens. Input points only attend to nearby latent tokens, which significantly boosts both efficiency and performance.
- Point-wise Training Objective: The model is trained to predict the velocity of individual points independently, conditioned on a global context. This effectively makes the model a "probabilistic extension of an INR," allowing it to be evaluated at any resolution during inference.
Key Insights and Performance
Because the model learns a continuous probability density field, it is inherently resolution-agnostic. For example, you can train INRFlow on 256x256 images but sample it at 2048x2048. The results remain sharp and coherent because the model operates in continuous space rather than discrete pixels.
Furthermore, the exact same transformer architecture and training objective achieved strong results across three distinct modalities, proving the viability of a "single recipe" generative engine:
- Images: On ImageNet-128, INRFlow achieved an FID of 2.73, outperforming other function-space models and remaining competitive with domain-specific diffusion models like ADM.
- 3D Generation: In image-to-3D tasks on the Objaverse dataset (800k 3D objects), it outperformed specialized baselines like CLAY and Michelangelo, achieving a ULIP-I score of 0.2976.
- Proteins: Tested on SwissProt using AlphaFold ground truths, it successfully generated physically plausible protein structures with a TM-score of 0.664, proving it can handle discrete sequences and complex 3D folding constraints.
The authors also demonstrated clear scaling laws: as model size increased from 35M parameters to 733M parameters, performance improved consistently across all metrics.
Limitations and Future Directions
While INRFlow beats other ambient space models, it is still slightly outperformed by latent-space models (like DiT) on specific image benchmarks. The authors attribute this "latent gap" to the massive, highly optimized datasets used to pre-train the VAEs that latent models rely on. Additionally, training directly in ambient space on very dense data (like high-resolution video) remains computationally expensive compared to operating in low-dimensional latent spaces.
Looking forward, the authors suggest exploring "co-training" across multiple modalities simultaneously. If a single model can learn a unified representation of the physical world—such as learning 3D geometry from both images and point clouds concurrently—it could pave the way for true general-purpose generative AI.
Summary updated