Introduction
Diffusion models have established themselves as the state-of-the-art for high-fidelity image and video synthesis. However, their iterative sampling process—often requiring hundreds of steps—remains a significant computational bottleneck. Consistency Distillation (CD) emerged as a powerful solution, training "student" models to map any point on a teacher model's probability flow ODE trajectory directly to the trajectory's origin. This allows for high-quality sampling in as few as one or two steps.
A pivotal but often overlooked element of these models is preconditioning. Preconditioning involves applying mathematical coefficients to the input data and network outputs to ensure the model satisfies essential boundary conditions. Historically, these coefficients were designed using intuition or empirical trial-and-error. This paper introduces Analytic-Precond, a theoretically grounded framework that replaces hand-crafted formulas with analytically optimized coefficients, significantly improving training efficiency and model alignment.
The Problem: Suboptimal Preconditioning
In Consistency Models (CM) and Consistency Trajectory Models (CTM), preconditioning ensures that as the time step $t$ approaches zero, the model's output converges to the original data. While existing formulas worked reasonably well for standard CMs—which primarily focus on "jumping" from noise ($t=T$) to data ($t=0$)—they proved highly suboptimal for CTMs. CTMs are designed to jump between any two points on the trajectory, and the lack of a rigorous mathematical foundation for preconditioning in these intermediate "jumps" led to slower training convergence and poor fidelity to the teacher model's path.
Methodology: The Analytic-Precond Framework
The authors propose Analytic-Precond, a principled approach to designing preconditioning coefficients by treating the problem as a form of ODE discretization. The method consists of three primary components:
- Generalized ODE Representation: The researchers reformulated the teacher's probability flow ODE into a generalized version incorporating free parameters ($l_t$ and $s_t$). This transformation allows preconditioning to be viewed through the lens of numerical ODE solvers.
- Minimizing the Consistency Gap: The "consistency gap" represents the discrepancy between the teacher's denoiser and the optimal student denoiser. By deriving analytical solutions that minimize a theoretical bound on this gap, the authors ensure the student model is initialized in a state that closely mirrors the teacher's behavior from the start of training.
- Backward Euler Stability: To maintain stability during the large "jumps" required by consistency models, the authors utilized a backward rewriting technique. This approach draws inspiration from the Backward Euler method, known in numerical analysis for its superior stability in solving stiff differential equations.
Notably, these optimized coefficients are computationally efficient; they can be calculated using automatic differentiation with an overhead of less than 1% of the total training time.
Experimental Results
The researchers evaluated Analytic-Precond across several standard benchmarks, including CIFAR-10, FFHQ (64×64), and ImageNet (64×64). The results demonstrate significant improvements over baseline consistency distillation methods.
Training Acceleration
The most striking result is the 2× to 3× acceleration in training convergence for multi-step consistency models. By using Analytic-Precond, models reached lower Fréchet Inception Distance (FID) scores—a key metric for image quality—much earlier in the training process than those using traditional preconditioning.
Trajectory Fidelity
Beyond speed, the method improves how closely the student model follows the teacher's "ground truth" trajectory. In 1D toy experiments, the authors visualized that Analytic-Precond produces paths that align more accurately with the teacher's ODE compared to previous hand-crafted methods. This suggests that the student model is learning a more faithful representation of the underlying data distribution.
Key Insights and Discussion
The paper provides a critical "aha moment" for the field: preconditioning is not merely a boundary-condition trick but a form of discretization that dictates how well a student model can approximate a teacher's trajectory.
While the original formulas used in Consistency Models were found to be near-optimal for single-step jumps to the origin, they were inadequate for the flexible, multi-point jumps required by Consistency Trajectory Models. Analytic-Precond bridges this gap. However, the authors note that while training is significantly faster, the final peak image quality (FID) after exhaustive training is only marginally improved. The primary value of this research lies in its efficiency and theoretical clarity rather than raising the absolute ceiling of generative performance.
Conclusion
Analytic-Precond moves consistency distillation away from empirical guesswork toward a rigorous analytical foundation. By optimizing the coefficients that govern how models "jump" across ODE trajectories, the authors have provided a path toward significantly faster and more stable training of high-performance generative models. This principled approach to trajectory jumping may have implications beyond image synthesis, potentially benefiting fields like robotics and control systems where efficient path approximation is critical.
Summary updated