TL;DR
eDMD (Endpoint Distribution Matching Distillation) significantly improves the quality of few-step diffusion models by training an auxiliary "fake" score model exclusively on the student's final outputs. By decoupling the distillation signal from intermediate denoising states, the authors eliminate a persistent mathematical bias (the "reference residual"), allowing 4-step models to achieve photorealistic results that rival their 40-step teachers.
What Problem Does This Paper Address?
Diffusion models have revolutionized image generation, but their iterative nature makes them slow, often requiring 30–50 network evaluations to produce a single high-quality image. Distillation aims to "compress" this process into just 1–4 steps.
A leading technique for this is Distribution Matching Distillation (DMD). In DMD, a "student" model is trained using two signals: a "teacher" model (the original slow model) and a "fake" model (an auxiliary model trained online to track the student's current distribution). The student is pushed toward the teacher and away from the "fake" distribution.
The authors identify a critical flaw in how multi-step DMD is currently handled. Existing methods train the "fake" model on intermediate denoising steps or a mixture of all steps. However, in a few-step student, the intermediate states don't necessarily need to follow the teacher's distribution—only the final "endpoint" image matters. Forcing intermediate steps to match a reference creates a "reference residual"—a mathematical ghost that continues to push the student away from the target even after it has technically converged.
Methods
The core of the paper is Endpoint Distribution Matching Distillation (eDMD), which introduces three key innovations:
- Endpoint Reference: Unlike previous methods that train the fake score model on all denoising steps, eDMD trains a single shared fake model exclusively on the student's final generated outputs. This ensures the correction signal vanishes exactly when the student's final output matches the teacher's distribution.
- Precise Teacher (PT): Standard distillation uses a single-step teacher prediction as a target. However, single-step predictions can be "blurry" because they average over many possible outcomes. eDMD uses a "Precise Teacher" that performs a short multi-step trajectory (e.g., 4 Euler steps) for the final denoising interval, providing a much sharper and more accurate target.
- Calibrated Query Density: Not all noise levels provide equally reliable supervision. The authors introduce a calibration step that estimates where the teacher model is most consistent across noise levels and concentrates training queries in those "reliable" zones.
Datasets
The authors evaluated eDMD across four major diffusion and flow-matching backbones:
- SD3-Medium
- FLUX.2-klein (4B and 9B versions)
- Qwen-Image-2.1
For training, they used a corpus of 191,500 prompts (compiled from T2I-2M, LAION, and others) and prompt-enhanced rewrites for the Qwen models. Evaluation was performed on DrawBench-1000 and DPG-Bench to measure human preference, aesthetic quality, and prompt adherence.
Key Insights and Ideas
- The Reference Residual: The paper's most significant theoretical contribution is proving that training on intermediate states introduces a non-zero residual. This residual can actually deflect the student away from the desired distribution.
- Intermediate Freedom: By only constraining the endpoint, intermediate steps are given the "freedom" to take whatever form best suits a 4-step generation process. On FLUX models, for instance, the authors found that intermediate predictions actually grow to nearly 3x the norm of the final output to achieve better results—a behavior that would be suppressed by traditional distillation.
- On-Tree vs. Off-Tree Targets: Figure 2 (bottom right) illustrates that single-step teacher calls often fall "off-tree" (between branches of the data distribution), while the Precise Teacher "walks" the teacher's ODE to stay "on-tree," leading to better fine-grained details like fur and text.
Key Results
eDMD consistently outperformed existing state-of-the-art distillation methods (like DMD2, TDM, and CDM) across all tested backbones:
- State-of-the-Art Performance: eDMD achieved the highest HPSv3++ (Human Preference Score) across all four backbones for 4-step generation.
- Visual Fidelity: As seen in Figure 1, the 4-step distilled Qwen-Image-2.1 model renders photorealistic scenes and legible text, which is notoriously difficult for few-step models.
- Convergence: In synthetic 2D experiments, the endpoint reference was the only method that allowed the student to perfectly match the teacher's complex tree-like distribution without being deflected by intermediate biases.
Limitations and Future Work
- Training Overhead: While eDMD doesn't increase inference cost, it does require training an auxiliary fake model online, which adds to the computational budget during the distillation phase.
- Teacher Dependence: The quality of the student is still capped by the teacher. If the teacher has inherent biases or artifacts, the student will likely inherit them.
- Step Count: While 4-step generation is a massive improvement over 40 steps, the industry continues to push toward 1-step "lightning" models. The authors suggest that the endpoint principle is even more critical as the step count decreases.
Future work could involve exploring how endpoint matching interacts with adversarial losses or applying these principles to video and audio distillation.
Summary updated