Asuka-FM: A System-Architectural Co-Design Framework for Efficient Text-to-Image Pretraining on Consumer Hardware

Víctor Rodrigo González, Pablo M. Olmos, Ignacio Peis, Alejandro LanchoarXiv 2026 · arXiv

TL;DR

Asuka-FM is an end-to-end framework that enables pretraining competitive text-to-image (T2I) models from scratch on a single consumer GPU (NVIDIA RTX 4090) in just ~160 hours for less than $55. By co-designing model architecture with system-level optimizations, it bypasses the massive compute barriers that typically restrict foundation model training to industrial-scale data centers.

What Problem Does This Paper Address?

The "accessibility gap" in generative AI is widening. State-of-the-art T2I models like FLUX.1 or SD3 boast billions of parameters and require thousands of H100 GPUs to train. For researchers with a single desktop workstation, three major bottlenecks make training from scratch nearly impossible:

  1. Quadratic Attention Costs: Standard transformers scale poorly with sequence length, wasting FLOPs on redundant spatial details.
  2. JIT Compilation Breaks: Modern compilers (like torch.compile) hate dynamic shapes. Standard "aspect-ratio bucketing" triggers constant kernel recompilations, tanking hardware utilization.
  3. The Autoencoder Wall: In Latent Diffusion Models, the Variational Autoencoder (VAE) is a massive throughput bottleneck. During training, VAE encoding can consume 60% of total time; at inference, decoding accounts for over 70% of latency in few-step regimes.

Methods

Asuka-FM introduces a 206M parameter dual-stream Diffusion Transformer (DiT) built on four pillars of efficiency:

  • Minimalist Backbone: It eliminates per-block Adaptive Layer Normalization (AdaLN) overhead, instead prepending a continuous sinusoidal timestep token directly to the multimodal sequence.
  • SPRINT (Sparse-Dense Residual Fusion): This technique partitions the 13-layer transformer into functional stages. While shallow and deep layers process full sequences, 9 "heavy" middle blocks use 75% sparse token dropping, slashing attention FLOPs by over 50% without losing macro-semantic coherence.
  • Continuous Aspect-Normalized RoPE: Instead of dynamic bucketing, Asuka-FM uses fixed square crops but conditions the model on continuous coordinates. This keeps tensor shapes static for the compiler while unlocking "zero-shot virtual camera control" (panning and zooming) at inference.
  • Latent-to-Pixel Adaptation: To break the "Autoencoder Wall," the authors transition the model from latent space to pixel space in a final training stage. They attach a lightweight (10M parameter) Patch Detailer Head (PDH) that maps latents directly to RGB pixels using $v$-prediction, effectively removing the VAE from the inference pipeline.
The multi-stage training roadmap of Asuka-FM, progressing from 256px base pretraining to 512px high-res adaptation and final pixel-space transfer.

Datasets

The model was trained on a balanced 625K multi-source open corpus including:

  • FLUX-Reason-6M (complex composition)
  • GPT-Edit-Simpler (instructional edits)
  • Midjourney-v6-Recap (artistic lighting)
  • Pexels (photography)
  • TextAtlas5M (text-rich scenes)

For domain-specific validation, they also curated a 337K Danbooru Corpus of high-density anime illustrations. Evaluation was performed using GenEval (for compositional alignment) and DPG-Bench.

Key Insights and Ideas

  • System-Architectural Co-Design: The biggest gains didn't come from just "making the model smaller," but from making the model "compiler-friendly." By enforcing static shapes through shifted square crops and continuous RoPE, they achieved a staggering 75.6% physical hardware utilization (MFU) on a consumer GPU.
  • The Amortization Asymmetry: The authors discovered that while removing the VAE is a huge win for pretraining and few-step inference (up to 3.34x speedup at 1 step), the recurrent nature of their pixel-space head makes it slightly slower than a standard VAE for high-step counts ($N \geq 13$).
  • Zero-Shot Camera Control: Because the model is conditioned on continuous coordinates rather than discrete indices, you can "steer" the virtual camera viewport at inference time by simply applying affine transformations to the coordinate mesh—no retraining required.
The Asuka-FM system-architectural co-design pipeline showing the SPRINT sparsification and decoupled pixel transfer.

Key Results

  • Extreme Efficiency: Asuka-FM sets a new Pareto-optimal standard. It achieves a GenEval of 0.609 and DPG-Bench of 78.51 at 512px resolution.
  • Cost-Effective: The entire latent pretraining pipeline (Stages 1 & 2) cost only **$49.17** in compute. Including the pixel adaptation, the total cost remains under $55.
  • Hardware Performance: On an RTX 4090, SPRINT enabled a throughput of 108.5 img/s (sparse) vs. 64.8 img/s (dense) at 512px, a 1.67x empirical speedup.
Figure 1: Asuka-FM framework overview. (a) Denoiser training compute vs. compositional alignment (DPG-Bench). (b) High-fidelity visual synthesis from our 206M dual-stream backbone. (c) Representative performance summary across industrial models, academic baselines, and Asuka-FM (†denotes finetuning on BLIP3o-60K).

Limitations and Future Work

  • Scale: While competitive, the 206M parameter backbone still trails multi-billion parameter models in rendering complex crowd scenes and extremely rare "long-tail" concepts.
  • Text Rendering: Despite including text-rich datasets, the model does not yet consistently render long, multi-word text strings perfectly.
  • Future Directions: The authors aim to scale the curriculum to native 1-megapixel (1024px) resolutions and leverage the "zero-decode" nature of the model to enable scalable online Reinforcement Learning (e.g., Flow-GRPO) on consumer hardware.
Sampling latency trade-off at 1MP: Pixel-space sampling is significantly faster in few-step regimes, while latent sampling becomes faster as the number of steps increases.

Summary updated

Abstract

Frontier text-to-image (T2I) foundation models predominantly rely on multi-billion parameter Diffusion Transformers (DiTs) pretrained across massive compute clusters, creating an accessibility barrier for academic laboratories. When training DiTs from scratch on single consumer GPUs (e.g., an NVIDIA RTX 4090 with 24\,GB VRAM), practitioners encounter three primary bottlenecks: quadratic attention sequence costs, Just-In-Time (JIT) compilation graph breaks induced by variable aspect-ratio bucketing, and an autoencoder throughput wall where frozen Variational Autoencoder (VAE) operations dominate step latency. In this work, we introduce \textbf{Asuka-FM}, an end-to-end system and architectural framework demonstrating that competitive dual-stream DiTs ($\sim$206M parameters) can be pretrained from scratch on a single consumer GPU in $\sim$160 GPU hours for under \$55. Asuka-FM integrates four core designs: (i) an AdaLN-free dual-stream DiT backbone reducing parameter count; (ii) structured sequence sparsification via Sparse-Dense Residual Fusion (SPRINT) to cut attention FLOPs by over 50\%; (iii) continuous aspect-normalized Rotary Position Embeddings (RoPE) coupled with shifted square crop training, guaranteeing static tensor shapes while unlocking zero-shot virtual camera control; and (iv) a pixel adaptation phase utilizing a lightweight (10M parameter) Patch Detailer Head operating under velocity regression ($v$-prediction). Evaluated on open-domain benchmarks without benchmark-specific fine-tuning, Asuka-FM achieves GenEval 0.588 / DPG 78.96 scores at 256px and GenEval 0.609 / DPG 78.51 scores at 512px, setting a new Pareto-optimal efficiency standard for democratic foundation modeling.
BibTeX
@misc{gonzalez2026asuka,
  author = {Víctor Rodrigo González and Pablo M. Olmos and Ignacio Peis and Alejandro Lancho},
  title = {{Asuka-FM: A System-Architectural Co-Design Framework for Efficient Text-to-Image Pretraining on Consumer Hardware}},
  year = {2026},
  eprint = {2611.26437},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26437},
}
Figure