Elucidating the Preconditioning in Consistency Distillation

Kaiwen Zheng, Guande He, Jianfei Chen, Fan Bao, Jun Zhu

TL;DR

This paper introduces Analytic-Precond, a principled method to mathematically optimize the "preconditioning" coefficients used in consistency distillation. By minimizing the "consistency gap" between teacher and student models, the authors achieve a 2x to 3x training speedup for multi-step generation in Consistency Trajectory Models (CTMs) across various image datasets.

What Problem Does This Paper Address?

Diffusion models are the current gold standard for high-quality image and video generation, but they are notoriously slow, often requiring hundreds of neural network evaluations to produce a single sample. Consistency Distillation aims to solve this by training a "student" model to mimic the trajectory of a "teacher" diffusion model, enabling high-quality generation in just one or two steps.

A crucial component of this process is preconditioning: a technique that combines the input data and the network's output using specific coefficients ($\alpha$ and $\beta$) to ensure the model respects boundary conditions (e.g., at time zero, the prediction should be the data itself). Until now, these coefficients have been hand-crafted based on intuition. The authors argue that these heuristic choices are suboptimal, particularly for Consistency Trajectory Models (CTMs), which need to "jump" between arbitrary points on the generation path.

Figure 1: Consistency distillation with preconditioning coefficients α, β.

Methods

The authors bridge the gap between deep learning heuristics and numerical analysis through three key steps:

  1. ODE Discretization Link: They demonstrate that preconditioning is fundamentally equivalent to choosing a specific discretization method for the teacher's Probability Flow (PF) Ordinary Differential Equation (ODE).
  2. Generalized ODE Framework: They derive a generalized version of the teacher ODE that incorporates free parameters ($l_t$ and $s_t$). This framework allows for a whole family of possible preconditionings.
  3. Consistency Gap Minimization: They define the "consistency gap" as the difference between the teacher's denoiser and the optimal student denoiser. By deriving a theoretical bound for this gap and minimizing it, they analytically solve for the optimal parameters.

The resulting Analytic-Precond method allows these coefficients to be computed directly from the teacher model. This process is highly efficient, adding less than 1% to the total training time.

Datasets

The researchers evaluated their method on several standard image generation benchmarks:

  • CIFAR-10: Both unconditional and class-conditional settings.
  • FFHQ 64×64: A dataset of high-quality human faces.
  • ImageNet 64×64: A large-scale, diverse dataset for conditional generation.
  • 1-D Toy Example: A simple Gaussian mixture used to visualize trajectory alignment.

Key Insights and Ideas

  • The "Aha" Moment on CMs: The authors found that for single-step generation (jumping from noise all the way to the start), the standard hand-crafted coefficients are actually very close to the analytical optimum. This explains why original Consistency Models (CMs) were already so effective.
  • The CTM Opportunity: The real suboptimality lies in intermediate jumps (e.g., jumping from step 10 to step 5). Standard CTM preconditioning fails to align well with the teacher's trajectory here, which slows down training.
  • Trajectory Alignment: By minimizing the consistency gap, the student model is better initialized. It starts the training process much closer to the "ground truth" teacher trajectory, leading to faster convergence.
Figure 4: Visualizations of the preconditioning coefficient g(t, s) for CTM, and for Analytic-Precond under different datasets.

Key Results

  • Training Acceleration: Analytic-Precond achieves a 2x to 3x speedup in training time for multi-step generation. For example, on ImageNet 64×64, the model reaches a high-quality FID score much earlier than the baseline CTM.
  • Superior Initialization: As seen in the training curves, the proposed method starts with a significantly lower FID (better quality) right at the beginning of training compared to standard methods.
  • Multi-step Gains: While single-step performance is similar to baselines, the gains become increasingly apparent as the number of function evaluations (NFEs) increases, indicating better alignment between the consistency function and the denoiser.
Figure 3: Training curves for two-step generation showing significant speedup across datasets.

Limitations and Future Work

  • Diminishing Returns at Convergence: While Analytic-Precond speeds up training significantly, the final image quality (FID) at the end of long training runs is only slightly better than the baseline.
  • Single-Step Performance: The method does not significantly improve single-step generation because, as the authors discovered, existing heuristics were already near-optimal for that specific case.
  • Broader Impact: Like all fast generative models, there are concerns regarding the potential for creating malicious content or amplifying biases present in the training data.
  • Future Directions: The authors suggest applying these insights to other types of ODE trajectories, such as those found in robotics path planning or control systems.

Summary updated

Abstract

Consistency distillation is a prevalent way for accelerating diffusion models adopted in consistency (trajectory) models, in which a student model is trained to traverse backward on the probability flow (PF) ordinary differential equation (ODE) trajectory determined by the teacher model. Preconditioning is a vital technique for stabilizing consistency distillation, by linear combining the input data and the network output with pre-defined coefficients as the consistency function. It imposes the boundary condition of consistency functions without restricting the form and expressiveness of the neural network. However, previous preconditionings are hand-crafted and may be suboptimal choices. In this work, we offer the first theoretical insights into the preconditioning in consistency distillation, by elucidating its design criteria and the connection to the teacher ODE trajectory. Based on these analyses, we further propose a principled way dubbed \textit{Analytic-Precond} to analytically optimize the preconditioning according to the consistency gap (defined as the gap between the teacher denoiser and the optimal student denoiser) on a generalized teacher ODE. We demonstrate that Analytic-Precond can facilitate the learning of trajectory jumpers, enhance the alignment of the student trajectory with the teacher's, and achieve $2\times$ to $3\times$ training acceleration of consistency trajectory models in multi-step generation across various datasets.
BibTeX
@inproceedings{zheng2025elucidating,
  author = {Kaiwen Zheng and Guande He and Jianfei Chen and Fan Bao and Jun Zhu},
  title = {{Elucidating the Preconditioning in Consistency Distillation}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=55pCDKiS8B},
}
Figure