A Better Critic Matters for Distribution Matching Distillation

Zongze Wu, Xianbang Wang, Zhengyang Geng, Mingdeng Cao, Zhoutong Zhang, Richard Zhang, Eli Shechtman, Ruiming CaoarXiv 2026 · arXiv

Abstract

Distribution Matching Distillation (DMD) trains a few-step student using the difference between a teacher's score and an online critic's estimate of the student distribution's score. We investigate what the critic needs, in capacity and in information, to provide a reliable training signal. On the capacity side, controlled experiments across seven SiT model sizes on ImageNet show that increasing teacher size while keeping a small critic fixed can degrade student quality and cause training collapse. Scaling the critic alongside the teacher, a strategy we call joint scaling, prevents collapse in our experiments and allows the student to benefit from larger teachers. On the information side, rather than forcing the critic to model a step-agnostic mixture of student outputs, we introduce a step-aware critic (STAC) that conditions on the student's generation step to estimate step-specific scores. Together, joint scaling and STAC consistently improve few-step distillation across class-conditional ImageNet generation, text-to-image generation with SDXL and FLUX.2, and text-to-video generation with Wan2.1.
BibTeX
@misc{wu2026better,
  author = {Zongze Wu and Xianbang Wang and Zhengyang Geng and Mingdeng Cao and Zhoutong Zhang and Richard Zhang and Eli Shechtman and Ruiming Cao},
  title = {{A Better Critic Matters for Distribution Matching Distillation}},
  year = {2026},
  eprint = {2610.42135},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.42135},
}

Loading related content...

Figure