TL;DR
Ambient Space Flow Transformers (ASFT) introduce a domain-agnostic, single-stage framework for training flow-matching generative models directly in data (ambient) space. By treating data as continuous coordinate-to-value maps and utilizing a conditionally independent point-wise objective within a PerceiverIO-style transformer, ASFT eliminates the need for modality-specific data compressors (like image VAEs or 3D PointNets) while naturally enabling zero-shot super-resolution and arbitrary point density generation.
What Problem Does This Paper Address?
Modern generative diffusion and flow matching pipelines almost universally rely on a two-stage recipe:
- Train a domain-specific autoencoder/compressor (e.g., VQGAN, VAE) to compress raw data into a compact latent space.
- Train a generative transformer (e.g., DiT) within that latent space.
While computationally convenient, this paradigm has major drawbacks:
- Tuning complexity & lack of end-to-end training: Hyperparameters like the autoencoder’s KL penalty or adversarial loss weights dramatically dictate whether downstream generative training succeeds or collapses.
- Modality silos: Every data domain requires custom compression architectures (e.g., 2D ConvNets for images, PointNets for 3D point clouds), preventing a truly unified generative training stack.
- Rigid discretizations: Discrete grids constrain outputs to fixed training resolutions.
The authors ask: Can we train flow-matching transformers directly in ambient space, across multiple modalities, in a single training stage without hand-crafted domain compressors?
Methods
ASFT views data instances as continuous mappings $f: x \to y$ from coordinates $x$ to values $y$:
- Images: Pixel coordinates $x \in \mathbb{R}^2$ map to RGB values $y \in \mathbb{R}^3$.
- 3D Point Clouds: Base reference positions $x \in \mathbb{R}^3$ map to 3D deformation vectors $y \in \mathbb{R}^3$.
Coordinate-Value Pairs (x, y) ──► Spatial-Aware Latents (z) ──► Self-Attention Blocks
│
Target Coordinate x* ────────────► Cross-Attention Decoder ◄────────────┘
│
Point-wise Velocity u_t(x*, y*)
1. Conditionally Independent CFM (CICFM) Objective
Standard flow-matching velocity fields can be decomposed point-wise because the forward noise perturbation process is spatially independent. Real-world data exhibits strong spatial correlations, so ASFT conditions point-wise predictions on a global latent code $z_{f_t}$: $$\mathcal{L}_{\text{CICFM}} = \mathbb{E}_{t, f, \epsilon} \left[ \left\| v_\theta(x_{f_t}, y_{f_t}, t \mid z_{f_t}) - u_t(x_f, y_f \mid \epsilon) \right\|_2^2 \right]$$ where $u_t$ specifies a standard rectified flow / linear interpolant trajectory.
2. Architecture: PerceiverIO with Spatially Aware Latents
- Encoder: Built upon PerceiverIO to process variable set sizes. To overcome the attention bottleneck of vanilla Perceiver models on large grids, latents are assigned "pseudo-coordinates." Latent tokens only cross-attend to neighboring coordinate-value inputs in ambient space before undergoing standard self-attention.
- Decoder: A target coordinate query cross-attends sequentially to intermediate and final encoder latent states (a multi-level decoding strategy) to independently predict its point-wise flow velocity.
- Efficient Sub-Sampling: Because decoding is point-wise, models do not need to reconstruct the full grid during training. Training decodes only a random subset (e.g., 8,192 pixels out of 65,536 for a $256 \times 256$ image), cutting training FLOPs by over 20%.
Datasets
- 2D Image Generation: FFHQ-256, LSUN-Church-256, and ImageNet ($128 \times 128$ and $256 \times 256$).
- 3D Point Cloud Generation: ShapeNet (Airplane, Chair, Car, and unconditional all-55 categories).
- Image-to-3D Generation: Objaverse (800k diverse 3D objects, conditioned on rendered views encoded with DINOv2 and Plücker ray embeddings).
Key Insights and Ideas
- Decoupling representation from resolution: Because the network evaluates velocities independently per point conditioned on $z_{f_t}$, query coordinates can be continuous. At inference time, one can query arbitrary coordinates to synthesize images at $512^2$, $1024^2$, or $2048^2$, or sample 100k 3D points from a model trained on only 2k points.
- Spatially localized latents resolve Perceiver bottlenecks: Vanilla PerceiverIO fails on high-resolution image generation (yielding an FID of 65.09 on LSUN-Church). Partitioning latent tokens by spatial locality drops the FID to 7.03.
- True domain agnosticism: The exact same architecture and loss recipe operate on 2D images and unstructured 3D point clouds—the only difference is swapping coordinate/value dimensionality.
Key Results
- Function Space Generative Models: On FFHQ-256, ASFT-L achieves a $\text{FID}_{\text{CLIP}}$ of 2.18 (vs. $\infty$-Diff's 3.87 and StyleGAN2's 2.35). On LSUN-Church-256, ASFT-L achieves 5.51 (vs. $\infty$-Diff's 10.36 and StyleGAN2's 6.21).
- ImageNet-256: ASFT-XL (733M parameters) achieves FID 3.74 / IS 228.8, outperforming ambient pixel-space models like ADM (10.94) and Simple Diffusion U-Net (3.76), while approaching hybrid models like HDiT (3.21) without using convolutional priors.
- 3D Point Clouds (ShapeNet): ASFT-B (108M parameters) outperforms the latent diffusion model LION (110M parameters) across the majority of metrics—e.g., achieving a 1-NNA EMD of 55.70% vs. 57.75% on the 55-category benchmark.
- Image-to-3D (Objaverse): ASFT outperforms two-stage models like CLAY on both point-cloud FID (0.36 vs. 0.99) and alignment (ULIP-I 0.2976 vs. 0.2066).
- Super-Resolution Fidelity: Directly sampling FFHQ images at $512 \times 512$ with ASFT trained on $256 \times 256$ yields an FID of 23.09, outperforming bicubic upsampling (24.34).
Limitations and Future Work
- Compute Gap with Two-Stage Latent Models: While competitive with pixel-space diffusion, ASFT still lags slightly behind pre-trained latent models like DiT-XL on ImageNet-256 (FID 3.74 vs. 2.27), partly because DiT leverages VAEs pre-trained on much larger datasets.
- Decoding Optimization Bottlenecks: Densely decoding too many points (e.g., 16,384) during training led to training instability and minor performance drops, limiting scaling efficiency during full-resolution supervision.
- Irregular Geometries: Spatial-aware latent clustering proved highly effective on regular 2D grids, but provided smaller gains on irregular 3D point clouds.
- Future Work: Exploring multi-modal co-training within a single checkpoint to perform cross-domain generation end-to-end.
Summary updated