Dog-Breed Text-to-image Flow Matching from Pretraining to Fine-Tuning on 4 Consumer GPUs

Reyhaneh Esmailizadeh, Xingjian Leng, Zhanhao Liang, Liang Zheng

TL;DR

MiniDog is a new framework that democratizes text-to-image (T2I) model training by enabling full pretraining and fine-tuning in under 75 minutes on just four consumer-grade NVIDIA 3090 GPUs. By combining efficient architectures like LightningDiT with advanced acceleration techniques like representation alignment (iREPA), the authors provide a complete, open-source pipeline for researchers and students to build their own generative models without needing a massive server farm.

What Problem Does This Paper Address?

The prevailing wisdom in AI is that training text-to-image diffusion models requires gargantuan compute resources—thousands of H100 GPUs and months of time. While true for "frontier" models like Stable Diffusion or FLUX, this high barrier to entry prevents beginners and academic labs from experimenting with the full training lifecycle.

The authors argue that this "compute wall" is a perception issue for those just starting out. They introduce MiniDog, a minimal yet functional project focused on a specific domain (dog breeds) to show that the entire pipeline—from data preparation to pretraining and supervised fine-tuning (SFT)—can be done efficiently on hardware many researchers already own.

Methods

MiniDog isn't about reinventing the wheel; it’s a "best-of" recipe for efficient training. The framework utilizes several key components:

  • Flow Matching & Rectified Flow: Instead of standard diffusion, they use flow matching, which learns a straight-line path between noise and data, making the sampling process more efficient.
  • Architecture: The core is a LightningDiT, a compact Diffusion Transformer with only 22 million parameters. It uses Qwen3-0.6B as a frozen text encoder and E2E-INVAE as the latent tokenizer.
  • iREPA (Improved Representation Alignment): This is the "secret sauce" for speed. It aligns the intermediate layers of the student model (the DiT) with a pretrained teacher model (a vision encoder like DINOv2 or EUPE). This helps the model learn spatial structures much faster by "peeking" at how a powerful vision model sees the image.
  • In-Context Conditioning: Timestep and text tokens are simply concatenated with image latents into a single sequence for the transformer, a method popularized by models like DiT and U-ViT.
  • Efficiency Hacks: To save time, all image and text features are precomputed and cached, meaning the GPUs only focus on updating the transformer weights during training.
Figure 2: T2I pretraining recipe. Frozen Qwen3 [14], E2E-INVAE [13] and EUPE ViT-B/16 [15] produce the text embeddings, the latents (noised with ϵ) and the alignment targets. LightningDiT [7] is pretrained with the flow-matching loss [2] and the iREPA [11] loss on Block 4.

Datasets

The project focuses on a niche but visually rich domain: dog breeds.

  • Pretraining: 26,000 images of 20 dog breeds sourced from ImageNet, recaptioned for better descriptive quality.
  • Fine-Tuning (SFT): 2,000 high-quality synthetic images generated by FLUX.1-schnell using prompts written by GPT-5.5. This demonstrates a modern trend: using larger, "teacher" generative models to create high-quality training data for smaller "student" models.
  • Evaluation: A held-out set of 500 prompts.

Key Insights and Ideas

  • Caption Length Matters: The authors found that training with longer captions (up to 128 tokens) leads to faster convergence and better image quality (lower FID) compared to short captions. Detailed descriptions provide a stronger learning signal.
  • The Choice of Teacher: For representation alignment, not all vision encoders are equal. The EUPE (Efficient Universal Perception Encoder) proved most effective for iREPA, likely due to its spatially coherent features.
  • MSE > Cosine: When aligning the student model to the teacher, using Mean Squared Error (MSE) loss consistently outperformed Cosine Similarity.
  • Tokenizer Quality is Crucial: The paper compares two VAEs (E2E-INVAE and E2E-VAVAE) and finds that the one with better reconstruction metrics also leads to a more "diffusable" latent space, making the generative model's job easier.
Figure 3: VAE diffusability compar- ison. Pretraining gFID with EUPE, iREPA, and 128 tokens.

Key Results

The efficiency of the MiniDog recipe is impressive:

  • Speed: On 4x RTX 3090 GPUs, pretraining takes 55 minutes (200 epochs) and SFT takes 15 minutes (100 epochs). Even on a single GPU, the whole process takes less than 4 hours.
  • Quality: The best pretraining recipe achieved a gFID of 8.32.
  • Human Preference: Supervised Fine-Tuning (SFT) significantly boosted performance. HPSv2 scores jumped from 0.1625 (pretrained) to 0.2427 (SFT), and PickScore rose from 18.42 to 19.61.

Qualitatively, the model produces distinct, recognizable dog breeds with varied poses and backgrounds, as seen in the cherry-picked samples below.

Qualitative samples from the SFT model showing various dog breeds.

Limitations and Future Work

The authors are transparent about the project's scope:

  • Domain Specificity: MiniDog is currently limited to dog breeds. It is not a general-purpose T2I model.
  • Model Scale: At 22M parameters, the model is tiny compared to industry standards, which limits its ability to handle complex, multi-object prompts.
  • Technical Novelty: The authors explicitly state they do not claim technical novelty; the contribution is the optimized, accessible framework and "recipe."
  • Future Directions: The repository is open-sourced to encourage the community to extend this to other domains or larger architectures using the same efficient training principles.

Summary updated

Abstract

We generally perceive that training text-to-image diffusion models requires heavy compute: it is true if we aim at a comprehensive frontier model. This understanding prevents people from training their own models when getting started in the field. We develop MiniDog, a framework that enables pretraining diffusion transformers in 55 minutes and fine-tuning in another 15 minutes on four 3090 GPUs. For data, we use 26k dog images from ImageNet and caption them for pretraining and use 2k images generated by FLUX.1-schnell for fine-tuning. We precompute the image and text features and cache them for efficiency. For architecture, we use E2E-INVAE for its semantic latent space, a small transformer with 22M parameters, a small Qwen model for text encoding, and in-context conditioning. For training, we use iREPA and an EUPE perception encoder for acceleration. For evaluation, our repository supports FID, HPSv2 score, PickScore, and various VAE reconstruction metrics.
BibTeX
@misc{esmailizadeh2026dog,
  author = {Reyhaneh Esmailizadeh and Xingjian Leng and Zhanhao Liang and Liang Zheng},
  title = {{Dog-Breed Text-to-image Flow Matching from Pretraining to Fine-Tuning on 4 Consumer GPUs}},
  year = {2026},
  eprint = {2610.18870},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.18870},
}
Figure