SD3.5-Flash: Distribution-Guided Distillation of Generative Flows

Hmrishav Bandyopadhyay, Rahim Entezari, Jim Scott, Reshinth Adithyan, Yi-Zhe Song, Varun JampaniSubmission to ICLR 2026 · Withdrawn

TL;DR

SD3.5-Flash distills modern rectified flow models (specifically Stable Diffusion 3.5 Medium) into ultra-fast 2-step and 4-step generators capable of running on consumer hardware—from gaming GPUs to iPhones. By reforming distribution matching with timestep sharing, introducing split-timestep fine-tuning, and eliminating memory-heavy text encoders, the framework delivers up to an ~18× speedup while matching or exceeding the teacher model’s generation quality.

What Problem Does This Paper Address?

Flow-matching and rectified flow architectures (such as SD3.5 and Flux) produce state-of-the-art visual quality and prompt fidelity, but at severe computational expense: they typically demand 25–50 sampling steps, 16+ GB of VRAM, and tens of seconds per generation.

Standard few-step distillation methods like Distribution Matching Distillation (DMD) struggle in rectified flow settings. DMD typically adds random noise to intermediate predictions to match teacher and student distributions; in flow trajectories, this artificial re-noising corrupts velocity field estimates and injects severe gradient noise, degrading image composition. Furthermore, forcing a compressed network to generate an image in just 2 to 4 steps creates a capacity bottleneck where models sacrifice text alignment to preserve visual aesthetics. Finally, the massive text encoders (such as T5-XXL) common in modern pipelines bloat memory and latency on consumer and mobile devices.

Methods

The authors introduce SD3.5-Flash, built on several algorithmic and architectural techniques:

[Pre-training] ──> Trajectory Guidance Loss (LTG)
                        │
[Distillation] ──> Distribution Matching + Timestep Sharing (evaluating on student trajectory)
               ──> Latent Adversarial Objective (using proxy MM-DiT features + MLP heads)
                        │
[Alignment]    ──> Split-Timestep Fine-Tuning (branching into t ∈ (0, 500] and (500, 1000], then fusing 3:7)
                        │
[Deployment]   ──> Drop T5-XXL (null embeddings) + 8-bit MM-DiT + 6-bit CoreML (Apple Neural Engine)
  • Trajectory Guidance Pre-training ($L_{\text{TG}}$): Distills the 50-step SD3.5 Medium teacher into an initial 4-step checkpoint by minimizing the trajectory distance along the teacher’s ODE path.
  • Timestep Sharing for Flow Distillation: Instead of computing KL-divergence gradients by re-noising generated samples to arbitrary timesteps, the authors compute scores strictly at the student's actual scheduled inference timesteps ($t_i^s$). This avoids out-of-trajectory velocity predictions and stabilizes training.
  • Split-Timestep Fine-Tuning: To expand model capacity without increasing inference footprint, training splits the student into two branches trained on disjoint timestep regimes ($t \in (0, 500]$ and $t \in (500, 1000]$) with exponential moving average (EMA) stabilization ($\beta = 0.99$). The two branches are then merged via weight interpolation (3:7 ratio), significantly boosting prompt adherence.
  • Flow Latent Adversarial Training: The proxy student acts as a multi-layer feature extractor. Lightweight MLP discriminator heads operate directly on flow latents (rather than pixel space) and are randomly reinitialized ($p = 0.005$) to prevent overfitting.
  • Pipeline Compression: By exploiting SD3.5's encoder dropout pre-training, the 4.7B-parameter T5-XXL is omitted entirely and substituted with null embeddings (relying only on CLIP-L and CLIP-G). The MM-DiT backbone is quantized to 8-bit for desktop/laptop cards and 6-bit CoreML for Apple Silicon (rewriting operations like RMSNorm for the Apple Neural Engine).

Datasets

  • Training Data: Synthetic image-text pairs generated by SD3.5 Large (8B) using 32 steps at CFG 4.0.
  • Evaluation Datasets:
  • MS-COCO (30K captions): Evaluated for zero-shot text-to-image quality using FID, CLIPScore, ImageReward, and Aesthetic Score.
  • GenEval: An object-focused compositional benchmark testing spatial, count, and attribute bindings.
  • Curated Human Evaluation Benchmark: 507 prompts (expert-crafted prompts + a subset of PartiPrompts) annotated across 124 evaluators.

Key Insights and Ideas

  • Re-noising breaks rectified flows: In standard diffusion, adding Gaussian noise is native to the SDE. In rectified flows, which define deterministic linear ODE paths between noise and data, adding arbitrary noise jumps off the trajectory, yielding corrupted velocity fields.
  • Decouple-and-merge relieves capacity strain: Image synthesis has distinct stages (early timesteps establish coarse semantic layout; late timesteps refine high-frequency texture). Branching the model by timestep domain lets each specialist optimize its parameters before merging back into a single checkpoint.
  • T5-XXL is expendable on edge devices: Because SD3.5 was pretrained with text-encoder dropout, the pipeline degrades gracefully when T5-XXL is dropped, shedding gigabytes of memory with minimal drop in standard benchmark scores.

Key Results

  • Generation Speed & Footprint:
  • On an NVIDIA RTX 4090, 4-step generation takes 0.58s (16-bit w/ T5) and 0.35s (8-bit w/o T5 at 768px).
  • On an Apple A17 iPhone (8 GB), the 6-bit model generates a 512px image in 3.25s and a 768px image in 8.32s.
  • Peak VRAM drops from 17.58 GiB (16-bit w/ T5) down to 6.61 GiB (8-bit w/o T5).
  • Benchmark Quality:
  • On GenEval, 4-step SD3.5-Flash achieves 0.70, surpassing the 50-step SD3.5 Medium teacher (0.64) and beating baselines like SDXL-Lightning (0.54) and NitroFusion (0.55).
  • Outperforms competitors in ImageReward (1.10 vs. teacher's 0.91).
  • Human Preference: In blind user studies, 4-step SD3.5-Flash was favored over SD3.5M-Turbo, SDXL-Lightning, SDXL-DMD2, SWD, and the 50-step teacher in overall visual quality.

Limitations and Future Work

  • Drop in Complex Text Alignment Without T5: While dropping T5-XXL slashes memory requirements, qualitative evaluation reveals degraded prompt fidelity on highly intricate, conditional compositional prompts.
  • 2-Step vs. 4-Step Tradeoff: Although the 2-step variant outperforms existing 2-step baselines like SANA-Sprint, it occasionally misses smaller prompt elements or introduces minor artifacts compared to the 4-step model.
  • High FID Baseline: Like other SD3.5-based models (e.g., SWD, SD3.5M-Turbo), FID scores on COCO remain around ~28–29, reflecting the teacher distribution’s baseline characteristics rather than distillation degradation.

Summary updated

Abstract

We present SD3.5-Flash, an efficient few-step distillation framework that brings high-quality image generation to accessible consumer devices. Our approach distills computationally prohibitive rectified flow models through a reformulated distribution matching objective tailored specifically for few-step generation. We introduce two key innovations: "timestep sharing" to reduce gradient noise and "split-timestep fine-tuning" to improve prompt alignment. Combined with comprehensive pipeline optimizations like text encoder restructuring and specialized quantization, our system enables both rapid generation and memory-efficient deployment across different hardware configurations. This democratizes access across the full spectrum of devices, from mobile phones to desktop computers. Through extensive evaluation including large-scale user studies, we demonstrate that SD3.5-Flash consistently outperforms existing few-step methods, making advanced generative AI truly accessible for practical deployment.
BibTeX
@misc{bandyopadhyay2025sd,
  author = {Hmrishav Bandyopadhyay and Rahim Entezari and Jim Scott and Reshinth Adithyan and Yi-Zhe Song and Varun Jampani},
  title = {{SD3.5-Flash: Distribution-Guided Distillation of Generative Flows}},
  year = {2025},
  eprint = {2509.21318},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.21318},
}
Figure