Coordinate In and Value Out: Training Flow Transformers in Ambient Space

Yuyang Wang, Anurag Ranjan, Joshua M. Susskind, Miguel Ángel Bautista

Introduction

Generative modeling has traditionally relied on a two-stage paradigm: first, compressing high-dimensional data into a compact latent space using specialized autoencoders (VAEs), and second, training a generative model within that latent space. While successful, this approach requires domain-specific inductive biases—such as convolutions for images or specialized "Evoformers" for proteins—limiting the development of a truly unified generative framework.

INRFlow proposes a "one-size-fits-all" recipe for generative modeling. By treating data as a continuous mapping from coordinates to values—known as Implicit Neural Representations (INRs)—and applying Flow Matching directly in the original "ambient" data space, INRFlow eliminates the need for complex, hand-crafted compressors.

Methodology: The INRFlow Framework

The core innovation of INRFlow lies in its domain-agnostic representation of data. Instead of viewing an image as a discrete grid of pixels, the model treats it as a function $f$ mapping coordinates $(x, y)$ to color values $(R, G, B)$. This formulation extends naturally to 3D point clouds (mapping coordinates to occupancy) and proteins (mapping sequences to 3D atomic coordinates).

Architecture and Flow Matching

INRFlow utilizes a PerceiverIO transformer architecture. To manage the computational complexity of high-resolution data, the authors introduce spatial-aware latents. Each latent token is assigned a "pseudo-coordinate" and performs cross-attention only with nearby data points, ensuring efficiency without sacrificing global context.

The INRFlow architecture (a) uses a PerceiverIO transformer to map coordinates and noisy values to a velocity field, enabling generation across diverse modalities including ImageNet (b), 3D objects (c), and proteins (d).

The model is trained using Flow Matching, which learns a deterministic velocity field to push Gaussian noise toward the data distribution along straight paths. This is complemented by a Conditionally Independent CFM (CICFM) loss, which allows the model to predict the value of any single point independently given the latents. This independence enables the model to be trained on random sub-samples and evaluated at any resolution.

Experiments and Results

The authors evaluated INRFlow across three distinct domains: 2D images, 3D point clouds, and protein structures.

Cross-Domain Success

INRFlow demonstrated competitive performance across all tested modalities:

  • Images: On ImageNet-128, the model achieved an FID of 2.73, outperforming several domain-specific diffusion models like ADM. It also set new benchmarks for "function space" models on FFHQ and LSUN-Church.
  • 3D Point Clouds: Using the Objaverse dataset, INRFlow successfully generated complex 3D objects from single-image prompts.
  • Proteins: The model produced physically plausible protein backbones from amino-acid sequences using the SwissProt dataset.
High-fidelity samples generated by INRFlow on the ImageNet-256 dataset, demonstrating the model's ability to capture complex textures and structures without a VAE.

Key Insights

A significant finding is the model's resolution agnosticism. Because it learns a continuous function, a model trained on 256x256 images can generate 2048x2048 images at inference time by sampling the learned density at a finer grain. Furthermore, the model exhibits strong scaling behavior, with sample quality improving consistently as the model size increases from 35M to 733M parameters.

Discussion

INRFlow represents a significant step toward a unified generative foundation model. By bypassing the latent space, it avoids the limitations of domain-specific compressors. However, training directly in ambient space remains computationally expensive. While sub-sampling techniques mitigate this, the total Gflops required are higher than latent-space counterparts. Future research will focus on multi-modal co-training to further leverage the domain-agnostic nature of the INRFlow architecture.

Summary updated

BibTeX
@inproceedings{wang2025coordinate,
  author = {Yuyang Wang and Anurag Ranjan and Joshua M. Susskind and Miguel Ángel Bautista},
  title = {{Coordinate In and Value Out: Training Flow Transformers in Ambient Space}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=qIzYWidWZH},
}
Figure