Coordinate In and Value Out: Training Flow Transformers in Ambient Space

Yuyang Wang, Anurag Ranjan, Joshua M. Susskind, Miguel Ángel BautistaSubmission to ICLR 2025 · Rejected

TL;DR

Ambient Space Flow Transformers (ASFT) introduce a domain-agnostic, single-stage framework for training flow-matching generative models directly in data (ambient) space. By treating data as continuous coordinate-to-value maps and utilizing a conditionally independent point-wise objective within a PerceiverIO-style transformer, ASFT eliminates the need for modality-specific data compressors (like image VAEs or 3D PointNets) while naturally enabling zero-shot super-resolution and arbitrary point density generation.

What Problem Does This Paper Address?

Modern generative diffusion and flow matching pipelines almost universally rely on a two-stage recipe:

  1. Train a domain-specific autoencoder/compressor (e.g., VQGAN, VAE) to compress raw data into a compact latent space.
  2. Train a generative transformer (e.g., DiT) within that latent space.

While computationally convenient, this paradigm has major drawbacks:

  • Tuning complexity & lack of end-to-end training: Hyperparameters like the autoencoder’s KL penalty or adversarial loss weights dramatically dictate whether downstream generative training succeeds or collapses.
  • Modality silos: Every data domain requires custom compression architectures (e.g., 2D ConvNets for images, PointNets for 3D point clouds), preventing a truly unified generative training stack.
  • Rigid discretizations: Discrete grids constrain outputs to fixed training resolutions.

The authors ask: Can we train flow-matching transformers directly in ambient space, across multiple modalities, in a single training stage without hand-crafted domain compressors?

Methods

ASFT views data instances as continuous mappings $f: x \to y$ from coordinates $x$ to values $y$:

  • Images: Pixel coordinates $x \in \mathbb{R}^2$ map to RGB values $y \in \mathbb{R}^3$.
  • 3D Point Clouds: Base reference positions $x \in \mathbb{R}^3$ map to 3D deformation vectors $y \in \mathbb{R}^3$.
Coordinate-Value Pairs (x, y) ──► Spatial-Aware Latents (z) ──► Self-Attention Blocks
                                                                        │
Target Coordinate x* ────────────► Cross-Attention Decoder ◄────────────┘
                                         │
                               Point-wise Velocity u_t(x*, y*)

1. Conditionally Independent CFM (CICFM) Objective

Standard flow-matching velocity fields can be decomposed point-wise because the forward noise perturbation process is spatially independent. Real-world data exhibits strong spatial correlations, so ASFT conditions point-wise predictions on a global latent code $z_{f_t}$: $$\mathcal{L}_{\text{CICFM}} = \mathbb{E}_{t, f, \epsilon} \left[ \left\| v_\theta(x_{f_t}, y_{f_t}, t \mid z_{f_t}) - u_t(x_f, y_f \mid \epsilon) \right\|_2^2 \right]$$ where $u_t$ specifies a standard rectified flow / linear interpolant trajectory.

2. Architecture: PerceiverIO with Spatially Aware Latents

  • Encoder: Built upon PerceiverIO to process variable set sizes. To overcome the attention bottleneck of vanilla Perceiver models on large grids, latents are assigned "pseudo-coordinates." Latent tokens only cross-attend to neighboring coordinate-value inputs in ambient space before undergoing standard self-attention.
  • Decoder: A target coordinate query cross-attends sequentially to intermediate and final encoder latent states (a multi-level decoding strategy) to independently predict its point-wise flow velocity.
  • Efficient Sub-Sampling: Because decoding is point-wise, models do not need to reconstruct the full grid during training. Training decodes only a random subset (e.g., 8,192 pixels out of 65,536 for a $256 \times 256$ image), cutting training FLOPs by over 20%.

Datasets

  • 2D Image Generation: FFHQ-256, LSUN-Church-256, and ImageNet ($128 \times 128$ and $256 \times 256$).
  • 3D Point Cloud Generation: ShapeNet (Airplane, Chair, Car, and unconditional all-55 categories).
  • Image-to-3D Generation: Objaverse (800k diverse 3D objects, conditioned on rendered views encoded with DINOv2 and Plücker ray embeddings).

Key Insights and Ideas

  • Decoupling representation from resolution: Because the network evaluates velocities independently per point conditioned on $z_{f_t}$, query coordinates can be continuous. At inference time, one can query arbitrary coordinates to synthesize images at $512^2$, $1024^2$, or $2048^2$, or sample 100k 3D points from a model trained on only 2k points.
  • Spatially localized latents resolve Perceiver bottlenecks: Vanilla PerceiverIO fails on high-resolution image generation (yielding an FID of 65.09 on LSUN-Church). Partitioning latent tokens by spatial locality drops the FID to 7.03.
  • True domain agnosticism: The exact same architecture and loss recipe operate on 2D images and unstructured 3D point clouds—the only difference is swapping coordinate/value dimensionality.

Key Results

  • Function Space Generative Models: On FFHQ-256, ASFT-L achieves a $\text{FID}_{\text{CLIP}}$ of 2.18 (vs. $\infty$-Diff's 3.87 and StyleGAN2's 2.35). On LSUN-Church-256, ASFT-L achieves 5.51 (vs. $\infty$-Diff's 10.36 and StyleGAN2's 6.21).
  • ImageNet-256: ASFT-XL (733M parameters) achieves FID 3.74 / IS 228.8, outperforming ambient pixel-space models like ADM (10.94) and Simple Diffusion U-Net (3.76), while approaching hybrid models like HDiT (3.21) without using convolutional priors.
  • 3D Point Clouds (ShapeNet): ASFT-B (108M parameters) outperforms the latent diffusion model LION (110M parameters) across the majority of metrics—e.g., achieving a 1-NNA EMD of 55.70% vs. 57.75% on the 55-category benchmark.
  • Image-to-3D (Objaverse): ASFT outperforms two-stage models like CLAY on both point-cloud FID (0.36 vs. 0.99) and alignment (ULIP-I 0.2976 vs. 0.2066).
  • Super-Resolution Fidelity: Directly sampling FFHQ images at $512 \times 512$ with ASFT trained on $256 \times 256$ yields an FID of 23.09, outperforming bicubic upsampling (24.34).

Limitations and Future Work

  • Compute Gap with Two-Stage Latent Models: While competitive with pixel-space diffusion, ASFT still lags slightly behind pre-trained latent models like DiT-XL on ImageNet-256 (FID 3.74 vs. 2.27), partly because DiT leverages VAEs pre-trained on much larger datasets.
  • Decoding Optimization Bottlenecks: Densely decoding too many points (e.g., 16,384) during training led to training instability and minor performance drops, limiting scaling efficiency during full-resolution supervision.
  • Irregular Geometries: Spatial-aware latent clustering proved highly effective on regular 2D grids, but provided smaller gains on irregular 3D point clouds.
  • Future Work: Exploring multi-modal co-training within a single checkpoint to perform cross-domain generation end-to-end.

Summary updated

Abstract

Flow matching models have emerged as a powerful method for generative modeling on domains like images or videos, and even on unstructured data like 3D point clouds. These models are commonly trained in two stages: first, a data compressor (\ie a variational auto-encoder) is trained, and in a subsequent training stage a flow matching generative model is trained in the low-dimensional latent space of the data compressor. This two stage paradigm adds complexity to the overall training recipe and sets obstacles for unifying models across data domains, as specific data compressors are used for different data modalities. To this end, we introduce Ambient Space Flow Transformers (ASFT), a domain-agnostic approach to learn flow matching transformers in ambient space, sidestepping the requirement of training compressors and simplifying the training process. We introduce a conditionally independent point-wise training objective that enables ASFT to make predictions continuously in coordinate space. Our empirical results demonstrate that using general purpose transformer blocks, ASFT effectively handles different data modalities such as images and 3D point clouds, achieving strong performance in both domains and outperforming comparable approaches. ASFT is a promising step towards domain-agnostic flow matching generative models that can be trivially adopted in different data domains.
BibTeX
@inproceedings{wang2025coordinate,
  author = {Yuyang Wang and Anurag Ranjan and Joshua M. Susskind and Miguel Ángel Bautista},
  title = {{Coordinate In and Value Out: Training Flow Transformers in Ambient Space}},
  booktitle = {International Conference on Machine Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=qIzYWidWZH},
}
Figure